PMLRML4H Findings Track Collection Machine Learning for Health (ML4H) 2023

# XAIQA

Joel<EMAIL_ADDRESS>
Ardavan<EMAIL_ADDRESS>
Hamid<EMAIL_ADDRESS>
Sanjit<EMAIL_ADDRESS>
Jeffrey<EMAIL_ADDRESS>
Jaime<EMAIL_ADDRESS>
Eran<EMAIL_ADDRESS>
Optum  Minnetonka  MN  USA

# XAIQA: Explainer-Based Data Augmentation for Extractive Question Answering

Joel<EMAIL_ADDRESS>
Ardavan<EMAIL_ADDRESS>
Hamid<EMAIL_ADDRESS>
Sanjit<EMAIL_ADDRESS>
Jeffrey<EMAIL_ADDRESS>
Jaime<EMAIL_ADDRESS>
Eran<EMAIL_ADDRESS>
Optum  Minnetonka  MN  USA

###### Abstract

Extractive question answering (QA) systems can enable physicians and
researchers to query medical records, a foundational capability for designing
clinical studies and understanding patient medical history. However, building
these systems typically requires expert-annotated QA pairs. Large language
models (LLMs), which can perform extractive QA, depend on high quality data in
their prompts, specialized for the application domain. We introduce a novel
approach, XAIQA, for generating synthetic QA pairs at scale from data
naturally available in electronic health records. Our method uses the idea of
a classification model explainer to generate questions and answers about
medical concepts corresponding to medical codes. In an expert evaluation with
two physicians, our method identifies $2.2\times$ more semantic matches and
$3.8\times$ more clinical abbreviations than two popular approaches that use
sentence transformers to create QA pairs. In an ML evaluation, adding our QA
pairs improves performance of GPT-4 as an extractive QA model, including on
difficult questions. In both the expert and ML evaluations, we examine trade-
offs between our method and sentence transformers for QA pair generation
depending on question difficulty.

###### keywords:

Language Models, Transformers, Question Answering, Synthetic Data,
Explainability

##  1 Introduction

Figure 1: XAIQA for generating synthetic QA pairs from medical documents and
augmenting the training data or prompts of extractive QA models. The shapes
without arrows indicate shape types in the flow diagram.

Many document-level search and decision support problems can be framed as
question answering (QA). Given a document context and question, an extractive
QA model identifies spans of text likely to contain answers. Applications in
healthcare include checking medical guidelines and policies against patient
charts, querying the medical history of a patient, and identifying cohorts for
clinical trials (Demner-Fushman et al., 2009; Patrick and Li, 2012). However,
annotated QA pairs are difficult to obtain, especially for medical text.
Clinical QA models can be trained on generic QA datasets such as SQuAD
(Rajpurkar et al., 2018) then applied to the medical domain, but performance
suffers (Soni et al., 2022). It is typically possible to pay clinicians to
annotate a small set of medical documents with QA pairs, but these small
datasets often focus on one medical speciality.

In addition to training ML models for medical document QA, large language
models (LLMs) such as GPT-4 (OpenAI, 2023) can perform clinical evidence
extraction using in-context learning (ICL) (Agrawal et al., 2022). Providing
few-shot examples of Q’s and A’s via ICL can improve the performance of LLMs
and tailor them to specific problem domains, but for each new domain,
different in-context examples are required (Dong et al., 2023).

We focus on generating clinically accurate extractive QA pair examples that
answer natural Q’s about patient conditions of similar form to Q: ”Does the
patient have $X$ in their medical history?” A: ”The patient has a history of
$X$.” Systems that can answer these types of Q’s can empower clinicians and
researchers to quickly surface conditions, procedures, and treatments from
patient medical histories.

Explainable artificial intelligence (XAI) focuses on explaining ML model
predictions. We propose XAIQA (Figure 1), a method to construct QA pairs from
information naturally associated with documents in the electronic health
record (EHR) such as past medical conditions, drugs, and procedures. XAIQA
pairs can be used to augment existing QA datasets or LLM prompts for ICL.
XAIQA exploits the natural symmetry between the above QA problem formulation
(Does $X$ apply?) and the problem of explaining document classifiers with a
text classification explainer.

Three main advantages of XAIQA are (1) scale: XAIQA can form QA pairs for as
many document, medical code pairs as exist in an EHR, (2) groundedness: the
answer spans come from real medical document text and are not abstractive, and
(3) the ability to produce non-keyword matches between Q’s and A’s. Our
experiments with physician annotators suggest XAIQA generates QA pairs of
greater semantic complexity - $2.2\times$ more semantic matches and
$3.8\times$ more clinical abbreviations - than natural baselines and provides
statistically significant performance improvements to GPT-4 as an extractive
QA model with about 5% absolute improvement in ROUGE (Lin, 2004) on hard
questions.

##  2 Related Work

The need for medical QA data to train QA systems for the healthcare setting
has spurned extensive research. Pampari et al. (2018) repurposed annotations
from medical documents in i2b2111 https://www.i2b2.org/NLP/DataSets/ to
generate Q’s by asking physicians about i2b2 entities and normalizing the Q’s
into templates, creating the emrQA dataset. Yue et al. (2021) found that A’s
in the relations subset of emrQA better correspond to human annotation of Q’s
than do answers to medications Q’s. For this reason, we used the relations
subset for automated evaluation of XAIQA. They also argued that because the
A’s in emrQA are frequently incomplete, exact match (EM) is a poor evaluation
metric. Furthermore, they found that most of the A’s in both the medications
and relations subsets contain the key phrase in the Q. This introduces the
need to focus on hard Q’s. Shinoda et al. (2021a) proposed a method for
identifying Q’s in QA datasets that have low lexical overlap with the
documents. We used their approach to create hard subsets of the emrQA
relations and mimicQA (Yue et al., 2021) datasets.

Another related area of research is question generation (Alberti et al., 2019;
Yue et al., 2021; Lehman et al., 2022). Our work differs from these approaches
in that we focused specifically on constructing grounded answers to questions
of the form ”does the patient have $X$?” One natural way to create QA pairs of
this form is with sentence transformers (Reimers and Gurevych, 2019), given
their speed, performance, and availability via the sentence-transformers222
https://github.com/UKPLab/sentence-transformers Python library. Nguyen et al.
(2022) used this approach with BERT (Devlin et al., 2019) to generate QA
pairs, and we compared our method to theirs. Instead of BERT as the sentence
encoder, we used ClinicalBERT Alsentzer et al. (2019) for its pretraining in
the biomedical and clinical domains and MPNET (Song et al., 2020) for its top
performance on sentence embedding benchmarks.333
https://www.sbert.net/docs/pretrained_models.html

Several ML systems for the healthcare domain have used medical code
descriptions to add information about medical concepts and their relationships
to ML models (Mullenbach et al., 2018; Vu et al., 2020; Yuan et al., 2022). We
took inspiration from these approaches.

As a text classification explainer, we used the Masked Sampling Procedure
(MSP) (Stremmel et al., 2022) for its state-of-the-art performance explaining
long medical document classifiers (details in Section A), but other explainers
could be used such as the Sampling and Occlusion algorithm (Jin et al., 2020)
or Grad-SHAP (Lundberg and Lee, 2017). We modeled our evaluation of synthetic
QA pairs on the work from Stremmel et al. (2022). While they compared MSP to
other XAI methods for text classification, we used physician evaluation to
compare MSP-generated QA pairs to those generated via sentence transformers.

##  3 Datasets

To generate QA pairs, we used discharge summaries from MIMIC-III (Johnson et
al., 2016) and data splits from prior work (Mullenbach et al., 2018) on
classifying International Classification of Disease (version nine) codes
(ICD-9s). Many subsequent papers have focused on the top 50 most common ICD-9s
from this dataset which limits the diversity of conditions used for the
classification task. We focused instead on ICD-9s occurring in at least 100 of
the 47,724 total discharge summaries from Mullenbach et al. (2018), resulting
in the same splits and 1,124 total medical condition labels. We call this
dataset “MIMIC-III $\geq$ 100.” We elected to use MIMIC-III over MIMIC-IV
(Johnson et al., 2023) because of the public availability of these data
splits.444 https://github.com/jamesmullenbach/caml-mimic

We used two QA datasets in our ML evaluations. The first is the emrQA
relations dataset consisting of medical document Q’s and A’s. We used a random
5% sample of the test set consisting of 10,835 QA pairs to measure the
performance of GPT-4 when provided synthetic QA pairs as in-context examples.
We also fine-tuned Longformer for extractive QA as in Li et al. (2022) on
subsets of the emrQA relations train set with the addition of synthetic QA
pairs, following the recommendation from Yue et al. (2021) (see Section B). We
then evaluated on the full emrQA test set.

The second QA dataset we used to test models with synthetic QA pairs is
mimicQA from Yue et al. (2021). This dataset was released by the authors to be
used as a test set only, since the dataset is small. It contains 1,287 QA
pairs, 975 clinician-verified pairs and 312 which are clinician-generated.

##  4 Methods

###  4.1 Explainer-Based Data Augmentation

Data: $d$, $h$

Result: augPairs

Function  _XAIQA(_$d$ , $h$, $C$, $E$_)_:

augPairs $\leftarrow$ [ ]

for  _$i=1$ to $n$_ do

$A_{i}$ $\leftarrow E(C,d_{i})$

for  _$j=1$ to $l$_ do

if  _$hasLabel(d_{i},h_{j})$_ then

question $\leftarrow description(h_{j})$

score $\leftarrow max(A_{i}[:,j])$

answerIdx $\leftarrow argMax(A_{i}[:,j])$

answer $\leftarrow s_{answerIdx}$

augPairs.append(question, answer, $d_{i}$, score)

end if

end for

end for

return augPairs

Algorithm 1 XAIQA: $d$ is a set of $n$ documents, $h$ a set of $l$ labels, $C$
a classifier, and $E$ an explainer. We assume the existence of a $hasLabel$
function that checks if a document is positive for a label and a $description$
function that returns the textual description of a label. $A_{i}$ is the
matrix of importance scores for each label $l$ produced by the explainer for
the $i$th document.

We present Algorithm 1 based on the intuition that whatever information in a
document is useful for classifying document labels can also be used to form QA
pairs, where the label descriptions form the Q’s and the predictive text in
the document form the A’s. Unsupervised approaches for generating such matches
typically maximize a similarity metric like cosine similarity. While this can
provide some semantic but not lexical matches between Q’s and A’s, it has a
critical limitation: Cosine similarity is maximized when the tokens in the Q
are the same as the tokens in the A. Our approach is not limited in this way,
as a classification algorithm will use whatever text in a document most
improves classification performance.

Algorithm 1 takes as input a set of $n$ documents
$d=[d_{1},d_{2},\dots,d_{n}]$ tokenized into $m$ sentences
$s=[s_{1},s_{2},\dots,s_{m}]$ having up to $l$ labels
$h=[h_{1},h_{2},\dots,h_{l}]$, a classifier $C$ which outputs label scores for
each document, and a sentence explainer $E$. Note that the number of sentences
$m$ will vary per document $d_{i}$ based on the content as will the value of
each label (either 1 or 0 for each $h_{j}$). $E$ uses $C$ to produce an
$m\times l$ matrix $A_{i}$ for each document $d_{i}$ by running inference on
perturbed versions of the sentences in $d_{i}$. $A_{i}$ contains importance
scores for each sentence, label pair and can be used to identify the most
important sentences for each label prediction. More generally, the explainer
can use any feature attribution method to assign importance scores to
sentences using the trained classifier.

In this way Algorithm 1 builds QA pairs by iterating through documents and
labels, collecting the labels as Q’s and the sentences with the highest scores
from the explainer as A’s along with the sentence scores. QA pairs are
produced for each document, label pair for which the label is positive and can
later be filtered by score. That is, after running Algorithm 1, $augPairs$ can
be sorted to select the top $r$ QA pairs and associated contexts. These can
then be used directly for model training, combined with other QA pairs, or
used for ICL.

In addition to Algorithm 1, we present a post-processing step (Algorithm 2)
based on initial results in Section 5.1 which uses ClinicalBERT to further
segment answer sentences. See Section C.2 for more information.

####  4.1.1 Generating QA Pairs

To train a classifier to extract medical concepts, we selected Longformer
(Beltagy et al., 2020) for its demonstrated performance on long medical
documents such as discharge summaries (Li et al., 2022) and fine-tuned it as
described in Section C.3 on the MIMIC-III $\geq$ 100 train set. See
performance in Section C.3. We then used MSP, Longformer, and Algorithm 1 to
generate QA pairs from the combined validation and test sets of MIMIC-III
$\geq$ 100\. We used the same combined dataset to generate QA pairs with the
sentence transformer models.

###  4.2 Evaluating QA Pairs

We evaluated synthetic QA pairs in an expert evaluation and by using them as
few-shot examples for GPT-4. We also tested adding these pairs to a training
dataset used to fine-tune Longformer. To show our method produces Q’s and A’s
pertaining to complex clinical relationships and not just keywords, physician
annotators counted non-keyword matches and abbreviations in the A spans in
addition to lexical matches. In the ML evaluation, we focused on whether
synthetic pairs improve LLM performance, particularly on hard questions.

As baselines, we tested ClinicalBERT (Alsentzer et al., 2019) and MPNET (Song
et al., 2020) as sentence transformers for embedding sentence, ICD description
pairs and selecting the pairs with the highest similarity score. For both
methods we used the mean token embedding over the CLS token based on results
from Reimers and Gurevych (2019). To create synthetic pairs for all methods,
we selected samples from among the top $r=5,914$ pairs with the highest score
(cosine similarity or explainer score), as this is equal to the size of the 1%
subset of emrQA and enabled us to test 1:1, 2:1, and 5:1 ratios of base to
synthetic data in the Longformer evaluation.

####  4.2.1 Physician Evaluation

Table 1: Selected examples from physician evaluation. In the question, we omit ”Does the patient have $X$ in their medical history?” Sem (correct answer which is not a string match, abbreviation, or negation), Lex (Lexical/string match), Abbr (Abbreviation), Algo (synthetic QA algorithm). Question |  Answer |  Sem |  Lex |  Abbr |  Algo
---|---|---|---|---|---
Unspecified hypothyroidism |  Levothyroxine 100 mcg Tablet Sig: One (1) Tablet PO DAILY (Daily). |  1 |  0 |  0 |  XAIQA
Esophageal reflux |  Past Medical History: End stage dementia with recurrent aspiration Osteoporosis GERD Social History: Lives in nursing home. |  0 |  0 |  1 |  XAIQA
Unspecified glaucoma |  Glaucoma 11. |  0 |  1 |  0 |  MPNET
Dysphagia, unspecified |  Swallowing difficulties. |  1 |  0 |  0 |  MPNET
Acute osteomyelitis involving ankle and foot |  Irrigation and debridement, extensive, to bone of left lower extremity degloving injury and grade 3A open tibia fibula fracture. |  1 |  0 |  0 |  CBERT

Two physicians worked independently to annotate the same set of 800 blinded QA
pairs. We included: 200 pairs generated by randomly selecting ICD descriptions
as Q’s and sentences from the MIMIC-III $\geq$ 100 discharge summaries as A’s,
200 sampled from the top $r$ pairs generated from this dataset using XAIQA,
and 400 generated using cosine similarity between ICD descriptions and
discharge summary sentences. Of the 400, 200 came from the top $r$ pairs from
ClinicalBERT as the sentence encoder and 200 from MPNET.

The physicians were asked to annotate correct A’s to the synthetic Q’s about
ICD diagnoses, A’s with direct string matches to the key clinical concepts in
the question, abbreviation matches between QA concepts, and negations. We
considered QA pairs with correct A’s that were not string matches to be
interesting semantic matches. Such QA pairs can add diversity to a train
dataset or prompt, providing information about complex relationships between
medical concepts. See the exact annotation instructions in Section C.4.

To combine results from the two annotators, we counted semantic matches as
those marked as neither lexical (string/keyword) matches nor abbreviations
according to each physician but marked as correct A’s by at least one. We
counted lexical matches and abbreviations if either physician marked a pair as
such. We used two sample T-Tests without assuming equal variances to compare
results from each method.

Table 2: The number (proportion) of correct answers according to at least one physician but not marked as lexical matches (Lex) or abbreviations (Abbr) by either physician are denoted as semantic matches (Sem). ClinicalBERT is denoted as CBERT. 200 synthetic QA pairs were provided from each algorithm. Algorithms which are significantly better ($\alpha=0.05$) than the next best algorithm are bolded. Algo | Sem | Abbr | Lex
---|---|---|---
XAIQA | 65 (0.325) | 45 (0.225) | 108 (0.540)
CBERT | 30 (0.150) | 12 (0.060) | 128 (0.640)
MPNET | 14 (0.070) | 2 (0.010) | 181 (0.905)
Random | 5 (0.025) | 2 (0.010) | 6 (0.030)

####  4.2.2 ML Evaluation

A benefit of our method and the sentence transformers approaches is that they
never show example A’s to the QA model that exceed a sentence in length. As
such, we evaluated all methods primarily on ROUGE-2 (we refer to it here as
ROUGE), which measures bi-gram recall for predicted A spans (Lin, 2004). We
looked additionally at F1 and EM scores, but because we never showed the model
more than one sentence in a synthetic QA pair, predicted A spans did not
trivially maximize ROUGE by containing entire passages (See predicted A
lengths in appendix Table 12). Furthermore, by aiming to maximize Rouge, we
privileged methods which fully answered each question.

To measure downstream performance of synthetic QA pairs, we included them as
few-shot examples in GPT-4 prompts and compared with zero-shot inference. For
each method, we sampled 10 QA pairs from the top $r$ pairs based on the
context length of the 8k GPT-4 model when providing 100 characters of context
to the left and right of each A span for each in-context example. We elected
to provide A’s with some limited context rather than entire documents based on
Liu et al. (2023) and our initial experiments. We found that providing the
entire document as context per in-context example made performance suffer
drastically compared to the zero-shot setting (see Table 7). Because example A
spans and test documents varied in length, we automatically reduced the
prompts for GPT-4 one QA pair at a time if the prompt exceeded the 8k context
window. Prompts and API version are provided in Section C.5.

####  4.2.3 Hard Subset Creation

Many of the examples in emrQA relations and other QA datasets are direct
string matches. We are interested in the ability of a model to extract
complete A’s to medical Q’s when the Q’s and A’s have low lexical overlap. At
runtime, one cannot know which method to select based on lexical overlap
between Q’s and A’s, so we instead computed query context overlap (QCLO) as in
Shinoda et al. (2021b). See the equation for QCLO in Section C.6.

##  5 Results

###  5.1 Physician Evaluation

As shown in Table 2, XAIQA produced significantly more semantic matches than
the baselines ($2.2\times$ more than ClinicalBERT with $p<0.001$). Our method
also identified significantly more abbreviations than both baselines
($3.8\times$ more than ClinicalBERT with $p<0.001$). MPNET found the most
lexical matches. We omitted negations from Table 2, as each method identified
10 or fewer, limiting statistical power. Section D.1 contains results from
individual physicians, examples of false positive QA pairs with physician
commentary, and motivation for Algorithm 2 to post-process XAIQA pairs before
providing them to QA models. Annotator agreement is reported in Table 10.

Table 1 depicts examples selected by the physicians following the blind
experiment to illustrate some relationships represented in the synthetic QA
pairs. For example, XAIQA recognized the semantic match between the use of a
medication (levothyroxine), and the presence of “unspecified hypothyroidism.”
This drug is prescribed for virtually no other condition. XAIQA also captured
many abbreviations, as in “esophageal reflux,” where the abbreviation “GERD”
is semantically equivalent (the acronym stands for “gastroesophageal reflux
disease”). MPNET found the lexical match “unspecified glaucoma” and
“glaucoma.” String matches of this kind were typical for MPNET, though we note
MPNET is capable of identifying some semantic matches, e.g., “dysphagia” is
difficulty swallowing. ClinicalBERT recognized that if “irrigation and
debridement, extensive, to bone” are required, this is very likely to reflect
“osteomyelitis (bone infection), involving ankle and foot.”

###  5.2 ML Evaluation

Table 3: Test set results for GPT-4 with synthetic QA pairs. XAIQA with ClinicalBERT (CBERT) used for post-processing denoted as PP. Exact match denoted as EM. Dataset |  Method |  ROUGE |  F1 |  EM
---|---|---|---|---
emrQA |  Zero-Shot |  0.667 [0.659 - 0.675] |  0.737 [0.730 - 0.743] |  0.491 [0.482 - 0.501]
CBERT |  0.697 [0.690 - 0.705] |  0.744 [0.737 - 0.750] |  0.514 [0.505 - 0.524]
MPNET |  0.696 [0.688 - 0.703] |  0.747 [0.740 - 0.753] |  0.518 [0.509 - 0.527]
XAIQA Raw |  0.703 [0.695 - 0.711] |  0.735 [0.728 - 0.741] |  0.509 [0.500 - 0.518]
XAIQA PP |  0.708 [0.701 - 0.716] |  0.746 [0.739 - 0.752] |  0.518 [0.509 - 0.527]
mimicQA |  Zero-Shot |  0.520 [0.497 - 0.543] |  0.600 [0.579 - 0.621] |  0.300 [0.275 - 0.326]
CBERT |  0.550 [0.528 - 0.573] |  0.606 [0.586 - 0.627] |  0.314 [0.289 - 0.340]
MPNET |  0.556 [0.532 - 0.580] |  0.613 [0.593 - 0.634] |  0.324 [0.298 - 0.349]
XAIQA Raw |  0.555 [0.531 - 0.579] |  0.598 [0.577 - 0.620] |  0.316 [0.290 - 0.342]
XAIQA PP |  0.562 [0.538 - 0.586] |  0.613 [0.593 - 0.633] |  0.309 [0.285 - 0.333]
Figure 2: ROUGE scores of GPT-4 with XAIQA pairs on emrQA relations. PP denotes post-processing. Figure 3: ROUGE scores of GPT-4 with XAIQA pairs on mimicQA. PP denotes post-processing. Table 4: Mean test ROUGE on hardest 5 and 10% of Q’s.  Dataset |  Method |  5% |  10%
---|---|---|---
emrQA |  Zero-Shot |  0.612 |  0.630
CBERT |  0.646 |  0.663
MPNET |  0.634 |  0.650
XAIQA Raw |  0.658 |  0.670
XAIQA PP |  0.650 |  0.668
mimicQA |  Zero-Shot |  0.410 |  0.408
CBERT |  0.440 |  0.440
MPNET |  0.443 |  0.443
XAIQA Raw |  0.458 |  0.460
XAIQA PP |  0.478 |  0.480

Figure 2 compares adding in-context examples from XAIQA to zero-shot
performance of GPT-4 on 10,835 emrQA relations test pairs as well as the
hardest 5, 10, 25, and 50% in terms of QCLO. Figure 3 shows the same for the
mimicQA test set (1,287 total pairs). Performance is reported as the mean of
the performance metric over 1,000 bootstrap samples drawn from the test set
with 95% confidence intervals.

XAIQA provides statistically significant performance improvements (confidence
intervals do not overlap) over zero-shot GPT-4 on the full emrQA relations
test set as well as the hardest 25 and 50% of examples. Test set size
decreases when subsetting to difficult examples, widening the confidence
intervals, but the pattern in the mean ROUGE score over bootstrap iterations
is clear on the hardest subsets of emrQA and mimicQA, where the performance
improvement is about 5%.

As shown in Table 3, XAIQA with post-processing outperforms sentence
transformers approaches in terms of ROUGE with the same F1 scores. Comparing
raw and post-processed XAIQA, we observe post-processing improves F1. MPNET
performance is strong in F1 and EM on the full datasets, consistent with the
results of the expert evaluation where MPNET found the most correct A spans,
despite most of these being easy examples. On the hardest 5 and 10% of Q’s,
XAIQA outperforms the sentence transformers approaches in terms of ROUGE
(Table 4). Section D.2.2 includes ROUGE and F1 plots for all hard subsets
tested.

Fine-tuning Longformer on different ratios of base to synthetic QA pairs
(appendix Figure 8) improves performance once the ratio is 2:1 base to
synthetic. There, MPNET offers significant gains. At a 5:1 ratio, XAIQA with
post-processing and ClinicalBERT outperform no synthetic data. We hypothesize
Longformer needs more base data to benefit from semantic relationships present
in the QA pairs from XAIQA. At all ratios, Longformer outperforms GPT-4,
underscoring the value of fine-tuning when data is abundant.

##  6 Discussion

We presented a novel method, XAIQA, to generate QA pairs that can inform
models of complex relationships between medical document Q’s and A’s. With
XAIQA, ML practitioners can generate targeted examples at scale for any
medical concept to add to an LLM prompt without the need for human annotation.
Our approach excels at introducing semantic and clinical abbreviation
relationships between Q’s and A’s with $2.2\times$ more semantic matches and
$3.8\times$ more clinical abbreviations compared to ClinicalBERT. XAIQA
significantly improves GPT-4 as an extractive QA model with up to 5%
improvement on hard questions.

Future work is required to analyze optimal base to synthetic ratios for both
LLM ICL and LM fine-tuning to help ML practitioners understand how to make the
most of synthetic QA data. Beyond our experiments presented here, we believe
XAIQA can turn any document-level classification dataset into an extractive QA
dataset, augmenting existing QA data or LLM prompts with grounded and
semantically complex answer spans.

## References

  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag.  Large language models are few-shot clinical information extractors.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1998–2022, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  10.18653/v1/2022.emnlp-main.130.  URL https://aclanthology.org/2022.emnlp-main.130.
  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins.  Synthetic QA Corpora Generation with Roundtrip Consistency, June 2019.  URL http://arxiv.org/abs/1906.05416.  arXiv:1906.05416 [cs].
  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott.  Publicly Available Clinical BERT Embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA, June 2019\. Association for Computational Linguistics.  10.18653/v1/W19-1909.  URL https://aclanthology.org/W19-1909.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The Long-Document Transformer.  _arXiv:2004.05150 [cs]_ , December 2020.  URL http://arxiv.org/abs/2004.05150.
  * Demner-Fushman et al. (2009) Dina Demner-Fushman, Wendy W. Chapman, and Clement J. McDonald.  What can natural language processing do for clinical decision support?  _Journal of Biomedical Informatics_ , 42(5):760–772, October 2009.  ISSN 1532-0480.  10.1016/j.jbi.2009.08.007.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv:1810.04805 [cs]_ , May 2019.  URL http://arxiv.org/abs/1810.04805.
  * Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui.  A Survey on In-context Learning, June 2023.  URL http://arxiv.org/abs/2301.00234.  arXiv:2301.00234 [cs].
  * Jin et al. (2020) Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren.  Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models.  Technical Report arXiv:1911.06194, arXiv, June 2020.  URL http://arxiv.org/abs/1911.06194.
  * Johnson et al. (2016) Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark.  MIMIC-III, a freely accessible critical care database.  _Scientific Data_ , 3(1):160035, May 2016.  ISSN 2052-4463.  10.1038/sdata.2016.35.  URL https://www.nature.com/articles/sdata201635.
  * Johnson et al. (2023) Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark.  MIMIC-IV, a freely accessible electronic health record dataset.  _Scientific Data_ , 10(1):1, January 2023.  ISSN 2052-4463.  10.1038/s41597-022-01899-x.  URL https://www.nature.com/articles/s41597-022-01899-x.  Number: 1 Publisher: Nature Publishing Group.
  * Lehman et al. (2022) Eric Lehman, Vladislav Lialin, Katelyn Edelwina Legaspi, Anne Janelle Sy, Patricia Therese Pile, Nicole Rose Alberto, Richard Raymund Ragasa, Corinna Victoria Puyat, Marianne Katharina Taliño, Isabelle Rose Alberto, Pia Gabrielle Alfonso, Dana Moukheiber, Byron Wallace, Anna Rumshisky, Jennifer Liang, Preethi Raghavan, Leo Anthony Celi, and Peter Szolovits.  Learning to Ask Like a Physician.  In _Proceedings of the 4th Clinical Natural Language Processing Workshop_ , pages 74–86, Seattle, WA, July 2022. Association for Computational Linguistics.  10.18653/v1/2022.clinicalnlp-1.8.  URL https://aclanthology.org/2022.clinicalnlp-1.8.
  * Li et al. (2022) Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, and Yuan Luo.  Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences, April 2022.  URL http://arxiv.org/abs/2201.11838.  arXiv:2201.11838 [cs].
  * Lin (2004) Chin-Yew Lin.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.  Lost in the Middle: How Language Models Use Long Contexts, July 2023.  URL http://arxiv.org/abs/2307.03172.  arXiv:2307.03172 [cs].
  * Loper and Bird (2002) Edward Loper and Steven Bird.  NLTK: The Natural Language Toolkit, May 2002.  URL http://arxiv.org/abs/cs/0205028.  arXiv:cs/0205028.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled Weight Decay Regularization.  _arXiv:1711.05101 [cs, math]_ , January 2019.  URL http://arxiv.org/abs/1711.05101.
  * Lundberg and Lee (2017) Scott Lundberg and Su-In Lee.  A Unified Approach to Interpreting Model Predictions.  _arXiv:1705.07874 [cs, stat]_ , November 2017.  URL http://arxiv.org/abs/1705.07874.
  * Mullenbach et al. (2018) James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein.  Explainable Prediction of Medical Codes from Clinical Text.  _arXiv:1802.05695 [cs, stat]_ , April 2018.  URL http://arxiv.org/abs/1802.05695.
  * Nguyen et al. (2022) Nhung Thi-Hong Nguyen, Phuong Phan-Dieu Ha, Luan Thanh Nguyen, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen.  SPBERTQA: A Two-Stage Question Answering System Based on Sentence Transformers for Medical Texts, June 2022.  URL http://arxiv.org/abs/2206.09600.  arXiv:2206.09600 [cs].
  * OpenAI (2023) OpenAI.  GPT-4 Technical Report, March 2023.  URL http://arxiv.org/abs/2303.08774.  arXiv:2303.08774 [cs].
  * Pampari et al. (2018) Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng.  emrQA: A Large Corpus for Question Answering on Electronic Medical Records.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2357–2368, Brussels, Belgium, October 2018. Association for Computational Linguistics.  10.18653/v1/D18-1258.  URL https://aclanthology.org/D18-1258.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  _arXiv:1912.01703 [cs, stat]_ , December 2019.  URL http://arxiv.org/abs/1912.01703.
  * Patrick and Li (2012) Jon Patrick and Min Li.  An ontology for clinical questions about the contents of patient notes.  _Journal of Biomedical Informatics_ , 45(2):292–306, April 2012.  ISSN 1532-0480.  10.1016/j.jbi.2011.11.008.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know What You Don’t Know: Unanswerable Questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia, July 2018. Association for Computational Linguistics.  10.18653/v1/P18-2124.  URL https://aclanthology.org/P18-2124.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics.  10.18653/v1/D19-1410.  URL https://aclanthology.org/D19-1410.
  * Shinoda et al. (2021a) Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa.  Can Question Generation Debias Question Answering Models? A Case Study on Question–Context Lexical Overlap.  In _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_ , pages 63–72, Punta Cana, Dominican Republic, November 2021a. Association for Computational Linguistics.  10.18653/v1/2021.mrqa-1.6.  URL https://aclanthology.org/2021.mrqa-1.6.
  * Shinoda et al. (2021b) Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa.  Can Question Generation Debias Question Answering Models? A Case Study on Question–Context Lexical Overlap.  In _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_ , pages 63–72, Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics.  10.18653/v1/2021.mrqa-1.6.  URL https://aclanthology.org/2021.mrqa-1.6.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  MPNet: Masked and Permuted Pre-training for Language Understanding.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 16857–16867. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/c3a690be93aa602ee2dc0ccab5b7b67e-Abstract.html.
  * Soni et al. (2022) Sarvesh Soni, Meghana Gudala, Atieh Pajouhi, and Kirk Roberts.  RadQA: A Question Answering Dataset to Improve Comprehension of Radiology Reports.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6250–6259, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.672.
  * Stremmel et al. (2022) Joel Stremmel, Brian L. Hill, Jeffrey Hertzberg, Jaime Murillo, Llewelyn Allotey, and Eran Halperin.  Extend and Explain: Interpreting Very Long Language Models.  In _Proceedings of the 2nd Machine Learning for Health symposium_ , pages 218–258. PMLR, November 2022.  URL https://proceedings.mlr.press/v193/stremmel22a.html.
  * Vu et al. (2020) Thanh Vu, Dat Quoc Nguyen, and Anthony Nguyen.  A Label Attention Model for ICD Coding from Clinical Text.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , pages 3335–3341, Yokohama, Japan, July 2020. International Joint Conferences on Artificial Intelligence Organization.  ISBN 978-0-9992411-6-5.  10.24963/ijcai.2020/461.  URL https://www.ijcai.org/proceedings/2020/461.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Yuan et al. (2022) Zheng Yuan, Chuanqi Tan, and Songfang Huang.  Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 808–814, Dublin, Ireland, May 2022. Association for Computational Linguistics.  10.18653/v1/2022.acl-short.91.  URL https://aclanthology.org/2022.acl-short.91.
  * Yue et al. (2021) Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon Lin, and Huan Sun.  CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering, December 2021.  URL http://arxiv.org/abs/2010.16021.
  * Zaheer et al. (2021) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big Bird: Transformers for Longer Sequences.  _arXiv:2007.14062 [cs, stat]_ , January 2021.  URL http://arxiv.org/abs/2007.14062.

##  Appendix A Further Discussion of Related Work

Yue et al. (2021) benchmarked BERT models for medical document QA, and Li et
al. (2022) trained sparse-attention LMs for long document QA. We used their
approach to fine-tune Longformer (Beltagy et al., 2020).

We selected MSP for its superior performance to the Sampling and Occlusion
algorithm (Jin et al., 2020), which demonstrated better performance than Grad-
SHAP (Lundberg and Lee, 2017) for explaining text classifiers. Additionally,
(Stremmel et al., 2022) demonstrate the runtime efficiency of MSP in the
context of long medical documents with fine-tuned sparse-attention LMs.

The core idea of the algorithm is that MSP recovers the text blocks which are
predictive of document labels by randomly masking a subset of text blocks over
many iterations and then computing the difference in the probabilities of each
label from the classifier when each text block is unmasked versus masked.
These differences are averaged over all masking iterations. Because text
blocks are masked jointly, MSP simulates cases where mulitple pieces of
information pertaining to document labels are included and excluded. This
joint masking approach also leads to runtime efficiencies over simply masking
each text block. We used sentence-level text blocks instead of fixed length
blocks to provide more natural answer spans to our synthetic questions.

##  Appendix B Additional Description of Datasets

Yue et al. (2021) found that only 5% of the total relations train set is
necessary to obtain the same level of performance as training on the full
dataset. We trained Longformer on up to 5% of the dataset for this reason. We
used the full validation set for early stopping after five epochs of no
improvement. We evaluated the Longformer models on the full emrQA relations
test set but randomly sampled the test set to evaluate the GPT-4 models given
the high cost of using the OpenAI API.

##  Appendix C Supplemental Methods

###  C.1 Explainer-Based Data Augmentation

###  C.2 Post-Processing

We used Algorithm 2 to produce more concise A’s by further splitting sentences
identified by the explainer and selecting the segment with the highest cosine
similarity to the medical concept question, balancing the idea of
classification-based and similarity-based A selection. Specifically, Algorithm
2 uses the explainer to identify predictive sentences and cosine similarity to
resolve long sentences with bullet points, numbered lists, or semi-colons, as
are often found in medical documents. When applying post-processing in our
experiments, we used ClinicalBERT to generate question and segment embeddings.

Given $augPairs$ from Algorithm 1 and a PunktSentenceTokenizer555
https://www.nltk.org/api/nltk.tokenize.PunktSentenceTokenizer.html from the
natural language toolkit (NLTK) Python library as $punkSentTok$, where the
tokenizer is initialized with the characters `’.’, ’?’, ’!’, ’•’, ’\`’, ’;’,
’0)’, ’1)’, ’2)’, ’3)’, ’4)’, ’5)’, ’6)’, ’7)’, ’8)’, ’9)’, Algorithm 2
iterates through a list of $r$ $augPairs$ and extracts the Q’s, A’s, contexts,
and scores. Answer sentences are split into shorter segments using
$punkSentTok$ to create segments on bullet points, numbered lists, and semi-
colons, as are often found in medical documents. These segments are then
embedded using a sentence embedding function $sentEmb$ which applies one of
the sentence encoders with mean pooling of the output tokens. Question and
answer segment embeddings are compared and the segment with the highest cosine
similarity to the question is selected and used in an updated version of
$augPairs$ which we refer to as $augPairsPP$, the output of Algorithm 2.

Data: augPairs, punkSentTok

Result: augPairsPP

Function  _XAIQAPP(_augPairs, punkSentTok_)_:

augPairsPP $\leftarrow$ [ ]

for  _$i=0$ to $r$_ do

question $\leftarrow augPairs[i][0]$

answer $\leftarrow augPairs[i][1]$

context $\leftarrow augPairs[i][2]$

score $\leftarrow augPairs[i][3]$

splitAnswers $\leftarrow punkSentTok.tokenize(answer)$

aEmb $\leftarrow sentEmb(splitAnswers)$

qEmb $\leftarrow sentEmb(question)$

topMatch $\leftarrow argMax(cosSim(aEmb,qEmb))$

answerPP $\leftarrow splitAnswers[topMatch]$

augPairsPP.append(question, answerPP, context, score)

end for

return augPairsPP

Algorithm 2 XAIQA post-processing: $augPairs$ is a list of $r$ (question,
answer, context, score) tuples produced by XAIQA. $punkSentTok$ is a sentence
tokenizer. We assume the existence of a sentence embedding function $sentEmb$
and a function $cosSim$ that compute cosine similarity between embeddings.

The sentence splitting heuristics of the post-processing algorithm could be
applied before training the classifier; however, for simplicity, we used the
NLTK sentence tokenizer (Loper and Bird, 2002) to generate sentences for all
methods, which, by default, does not split on bulleted or numbered lists.

###  C.3 Generating QA Pairs

####  C.3.1 Fine-Tuning Longformer for Classification

We fine-tuned Longformer with a 4,096 token maximum sequence length for multi-
label classification using PyTorch (Paszke et al., 2019) (version 1.12.1) and
transformers version 4.22.1 (Wolf et al., 2020).

We used an effective batch size of 128 (two samples per GPU with eight
gradient accumulation steps on an Azure Standard_ND40rs_v2 machine with 32 GB
GPU RAM per GPU and eight GPUs in total) and the AdamW (Loshchilov and Hutter,
2019) optimizer with 0.01 weight decay, 500 warm-up steps and early stopping
after five evaluations (occurring every 50 steps) with no improvement in
validation loss.

####  C.3.2 Performance of Longformer Classifier

Table 5: Performance of LMs fine-tuned for multi-label classification on MIMIC-50. Model |  Micro-Average Precision |  Macro-Average Precision
---|---|---
BigBird Base (Raw Text) |  0.584 [0.572 - 0.595] |  0.4763 [0.466 - 0.487]
BigBird Base (Clean Text) |  0.659 [0.648 - 0.670] |  0.5526 [0.541 - 0.564]
BigBird Large (Raw Text) |  0.629 [0.618 - 0.641] |  0.5249 [0.514 - 0.536]
BigBird Large (Clean Text) |  0.698 [0.687 - 0.708] |  0.6003 [0.590 - 0.611]
Longformer Base (Raw Text) |  0.640 [0.629 - 0.652] |  0.5342 [0.524 - 0.545]
Longformer Base (Clean Text) |  0.652 [0.641 - 0.664] |  0.5529 [0.542 - 0.563]

We compared several classifiers to use for the MSP explainer on the MIMIC-50
dataset from Mullenbach et al. (2018), including base and large versions of
Longformer and Big Bird (Zaheer et al., 2021) with and without cleaning the
text of the discharge summaries using the logic from Stremmel et al. (2022).
Performance on the 1,730 test documents from MIMIC 50 are reported in Table 5.
All pretrained model checkpoints were taken from the Hugging Face model hub
666 https://huggingface.co/models .

Based on the MIMIC-50 experiments, we selected the Longformer base classifier
for it’s performance without text cleaning, enabling easier downstream
application of the model. We fine-tuned Longformer base on “MIMIC-III $\geq$
100” resulting in the performance in Table 6.

Table 6: Performance of Longformer base for multi-label classification on MIMIC-III $\geq$ 100. Model |  Micro-Average Precision |  Macro-Average Precision
---|---|---
Longformer Base (Raw Text) |  0.461 [0.4553- 0.467] |  0.221 [0.216 - 0.226]

We expect that higher performing multi-label classifiers would yield better
synthetic QA pairs using XAIQA and leave these experiments for future work.

###  C.4 Physician Evaluation

The physicians were asked to annotate the following four attributes for each
example:

  1. 1.

Correct answers to the synthetic questions about ICD diagnoses where the
question is whether the patient has the medical condition: This required the
correct form of a disease. For example, if a specific disease form was posed
in the question field, that form should be supported in the answer field. If
the question was for a general form, then a specific form in the answer field
which implied the general form or a general answer would be sufficient.

  2. 2.

Direct string matches between clinical concepts in the questions and clinical
concepts in the answers: This field was marked if the example was a correct
answer and the evidence included the same key clinical words as the diagnosis.
Casing, misspelling, varying forms of a word (e.g.,’ing’), and punctuation did
not matter. Anything very close to a string match was considered a string
match, though the exact interpretation of close was left to each physician’s
judgment. The idea was to record ”easy” matches.

  3. 3.

Abbreviation matches between question and answer concepts: This field was
marked if the example was a correct answer and the key clinical words in the
answer text were an abbreviated form of the diagnosis.

  4. 4.

Negations: This field was marked if the example was a correct answer and the
diagnosis was refuted by the provided text.

###  C.5 ML Evaluation

####  C.5.1 Performance with Entire Documents In-Context

Table 7: emrQA test set results for GPT-4 with synthetic QA pairs from MPNET and XAIQA. We tested both approaches using the full document as context with 5 in-context examples and a window of 100 characters to the left and right of the answer as context with 10 in-context examples. Method |  Context |  ROUGE |  F1 |  EM
---|---|---|---|---
MPNET |  Full |  0.483 [0.465 - 0.501] |  0.581 [0.564 - 0.597] |  0.309 [0.290 - 0.328]
XAIQA Raw |  Full |  0.515 [0.497 - 0.534] |  0.583 [0.565 - 0.600] |  0.347 [0.327 - 0.367]
MPNET |  Window |  0.696 [0.688 - 0.703] |  0.747 [0.740 - 0.753] |  0.518 [0.509 - 0.527]
XAIQA Raw |  Window |  0.703 [0.695 - 0.711] |  0.735 [0.728 - 0.741] |  0.509 [0.500 - 0.518]

####  C.5.2 GPT-4 Prompts

Table 8: Prompts for GPT-4 Setting |  Prompt
---|---
Zero-Shot |  Extract a span of text from the medical document to answer the supplied medical question with evidence from the document.

For example, if you are given the following basic example:

Question: ”Does the patient have hypertension”
Document: ”John Smith is a 56 year-old male. He was diagnosed with
hypertension on 04-04-22. John is at risk for stroke.”

You should answer with:

{”start_idx”: 35, ”span_text”: ”He was diagnosed with hypertension on
04-04-22.”}

Answer questions using only text found in the document. REMEMBER: Return only
properly formatted JSON. Do not return any additional text. You are an expert
in producing perfectly formatted JSON responses and no additional text. You
always remember to close bracks when outputting JSON. You are an expert in
medicine and medical document review. You provide perfect answers to questions
about patients by carefully reading their medical documents and using your
clinical knowledge.

Few-Shot |  Extract a span of text from the medical document to answer the supplied medical question with evidence from the document.

For example, if you are given the following basic example:

Question: ”Does the patient have hypertension”
Document: ”John Smith is a 56 year-old male. He was diagnosed with
hypertension on 04-04-22. John is at risk for stroke.”

You should answer with:

{”start_idx”: 35, ”span_text”: ”He was diagnosed with hypertension on
04-04-22.”}
Answer questions using only text found in the document. REMEMBER: Return only
properly formatted JSON. Do not return any additional text. You are an expert
in producing perfectly formatted JSON responses and no additional text. You
are an expert in medicine and medical document review. You provide perfect
answers to questions about patients by carefully reading their medical
documents and using your clinical knowledge.

Here are some additional examples:
…

Prompts for GPT-4 were as shown in Table 8.

For the few-shot prompts, after including examples, we provided the suffix:
”Now that you’ve seen some examples of what to do, extract a span of text from
the medical document to answer the supplied medical question with evidence
from the medical document. Answer questions using only text found in the
document. REMEMBER: Return only properly formatted JSON. Do not return any
additional text. You are an expert in producing perfectly formatted JSON
responses and no additional text. You always remember to close brackets when
outputting JSON. You are an expert in medicine and medical document review.
You provide perfect answers to questions about patients by carefully reading
their medical documents and using your clinical knowledge.”

We found that GPT-4 tended to ”forget” the original instructions when
providing few-shot examples if not reminded at the end using a suffix like
this.

For all experiments with GPT-4, we used the 03-15-2023 version of the OpenAI
API, the 8k context version of GPT-4, and temperature 0 to encourage
extractive answers.

####  C.5.3 Longformer Fine-Tuning for QA

We fine-tuned Longformer base for question answering using the code and
instructions from Li et al. (2022) documented here 777
https://github.com/luoyuanlab/Clinical-
Longformer/tree/main/Question%20Answering . We used PyTorch (Paszke et al.,
2019) (version 1.12.1) and transformers version 4.21.1 (Wolf et al., 2020) and
fine-tuned at a maximum sequence length of 4,096 tokens with an effective
batch size of 128 (two samples per GPU with eight gradient accumulation steps
on an Azure Standard_ND40rs_v2 machine with 32 GB GPU RAM per GPU and eight
GPUs in total) and the AdamW (Loshchilov and Hutter, 2019) optimizer with 0.01
weight decay, 10,000 warm-up steps and early stopping after 10 evaluations
(occurring every epoch) with no improvement in validation F1.

###  C.6 Hard Subset Creation

Shinoda et al. (2021b) define query context overlap as $QCLO=|Q\cap C|/|Q|$.
Here, Q denotes the set of words in the query, C, the set of words in the
context, and QCLO the ratio of intersecting words in Q and C to the number of
words in Q. Because this metric counts stopwords and question words which may
appear in the context as overlapping and does not count near matches for the
same word with a different suffix, e.g., ”ing”, we removed stopwords and
question words and applied the NLTK PorterStemmer888
https://tartarus.org/martin/PorterStemmer/ to the set of words in Q and C
before computing QCLO.

##  Appendix D Supplemental Results

###  D.1 Physician Evaluation

####  D.1.1 (

Individual Annotator Results)

Table 9: Number (proportion) of semantic but not lexical matches or abbreviations (Sem), abbreviations (Abbr), and lexical matches (Lex) identified in the expert evaluation. ClinicalBERT denoted as CBERT. 200 synthetic QA pairs were provided from each algorithm. Algorithms which are significantly better ($\alpha=0.05$) than the next best algorithm according to two-sample T-tests without assuming equal variances are bolded. | Reviewer 1 | Reviewer 2
---|---|---
Algo | Sem | Abbr | Lex | Sem | Abbr | Lex
XAIQA | 65 (0.325) | 44 (0.220) | 102 (0.510) | 79 (0.395) | 37 (0.185) | 74 (0.370)
CBERT | 25 (0.630) | 8 (0.040) | 126 (0.310) | 73 (0.365) | 11 (0.055) | 62 (0.310)
MPNET | 14 (0.070) | 1 (0.005) | 178 (0.890) | 53 (0.265) | 1 (0.005) | 129 (0.645)
Random | 3 (0.015) | 0 (0.000) | 5 (0.025) | 6 (0.030) | 2 (0.010) | 3 (0.015)

As shown in Table 9, XAIQA produced more semantic matches than the baselines
according to both reviewers, including significantly more than the second best
method according to reviewer 1 with $2.6\times$ more and ($P<0.001$). Our
method also identified significantly more abbreviations than both baselines
according to both reviewers, about $5.5\times$ more than the second best
method, ClinicalBERT, according to reviewer 1 and $3.4\times$ more according
to reviewer 2. MPNET found the most lexical matches according to both
reviewers.

####  D.1.2 Annotator Agreement

Table 10: Overall agreement and Cohen’s Kappa for each field annotated. Field |  Agreement |  Cohen’s Kappa
---|---|---
Correct Answer |  0.904 |  0.796
String Match |  0.791 |  0.586
Abbreviation |  0.978 |  0.815
Negation |  0.984 |  0.228

####  D.1.3 Example Synthetic QA Pairs

####  D.1.4 Motivation for Explainer Post-Processing

XAIQA approach was more likely to recover sections from the discharge
summaries such as History of Present Illness and Past Medical History. For
example, the past medical history section is recovered in about 34% of the top
synthetic QA pairs generated by the method. Cosine similarity, on the other
hand, only does this about 7% of the time with ClinicalBERT and less than 1%
of the time with MPNET. We hypothesize this is because cosine similarity is
reduced when sentences contain several clinical concepts. While recovering
this section when it contains the correct A is ideal when presenting
information to a human - it provides useful context - and not a problem for
expert evaluation, this finding motivates Algorithm 2 for post-processing.

####  D.1.5 Example False Positive Synthetic Pairs

Table 11: Selected false positive examples from physician evaluation. In the question, we omit ”Does the patient have $X$ in their medical history?” Sem (correct answer which is not a string match, abbreviation, or negation), Lex (Lexical/string match), Abbr (Abbreviation), Algo (synthetic QA algorithm). Question |  Answer |  Sem |  Lex |  Abbr |  Algo
---|---|---|---|---|---
Venous catheterization, not elsewhere classified |  He did develop ARDS and a ventilator associated pneumonia for which Vancomycin, Zosyn and Ciprofloxacin were started |  0 |  0 |  0 |  XAIQA
Pneumonia due to escherichia coli [E. coli] |  Klebsiella Pneumoniae. |  0 |  0 |  0 |  CBERT

Despite being pretrained on clinical text, ClinicalBERT made no shortage of
mistakes (49 in total out of 200 for annotator 1 and 65 in total for annotator
2). In Table 11, we show how ClinicalBERT incorrectly associated “… pneumonia
due to Escherichia coli” with “Klebsiella pneumoniae,” the genus and species
of a different bacterium whose species designator (“pneumoniae”) misled the
model into a pneumonia diagnosis, likely as a result of overlapping subtokens
when representing “pneumoniae” and pneumonia. Like the other approaches, XAIQA
produced some false positive predictions (33 in total out of 200 for annotator
1 and 47 in total for annotator 2), as in the question of ”venous
catheterization,” where the model incorrectly provided an answer related to
ARDS, a bad respiratory failure episode requiring a ventilator and blood draws
or indwelling IV lines. This correlation could account.

###  D.2 ML Evaluation

####  D.2.1 Predicted Answer Lengths

Table 12: 5th, 50th and 95th percentile of predicted answer lengths (in
tokens) for GPT-4 on emrQA relations.

Method | 5th | 50th | 95th
---|---|---|---
Zero-Shot | 2 | 8 | 18
CBERT | 2 | 10 | 22
MPNET | 2 | 9 | 19
XAIQA Raw | 2 | 10 | 24
XAIQA PP | 3 | 10 | 22

####  D.2.2 All Methods on Hard Subsets

We include plots of ROUGE score for the GPT-4 and Longformer experiments on
the hard subsets of both test datasets for all methods in Figure 4, Figure 5,
and Figure LABEL:fig:longformer_bar_results_ROUGE_base.

F1 score for QA balances n-gram recall and precision, rewarding models with
complete predicted answers that are no longer than necessary. We report F1s
for the GPT-4 and Longformer experiments in Figure 6, Figure 7, and Figure 9.

Figure 4: ROUGE scores of GPT-4 for extractive QA using different synthetic QA
pairs on hard subsets of emrQA relations. CBERT is ClinicalBERT. Figure 5:
ROUGE scores of GPT-4 for extractive QA using different synthetic QA pairs on
hard subsets of mimicQA. CBERT is ClinicalBERT. Figure 6: F1 scores of GPT-4
for extractive QA using different synthetic QA pairs on hard subsets of emrQA
relations. CBERT is ClinicalBERT. Figure 7: F1 scores of GPT-4 for extractive
QA using different synthetic QA pairs on hard subsets of mimicQA. CBERT is
ClinicalBERT Figure 8: ROUGE scores of Longformer fine-tuned using varying
ratios of base to synthetic pairs on emrQA relations. CBERT is ClinicalBERT.
Figure 9: F1 scores of Longformer fine-tuned for extractive QA using varying
ratios of base to synthetic QA pairs on emrQA relations. CBERT is
ClinicalBERT.


# Consistency and Interpretation of the LHC (Di-)Di-Jet Excesses

Andreas Crivellin<EMAIL_ADDRESS>Physik-Institut, Universität
Zürich, Winterthurerstrasse 190, CH–8057 Zürich, Switzerland  Paul Scherrer
Institut, CH–5232 Villigen PSI, Switzerland  Claudio Andrea Manzari
<EMAIL_ADDRESS>Physik-Institut, Universität Zürich,
Winterthurerstrasse 190, CH–8057 Zürich, Switzerland  Paul Scherrer Institut,
CH–5232 Villigen PSI, Switzerland  Bruce Mellado<EMAIL_ADDRESS>School
of Physics and Institute for Collider Particle Physics, University of the
Witwatersrand, Johannesburg, Wits 2050, South Africa  iThemba LABS, National
Research Foundation, PO Box 722, Somerset West 7129, South Africa  Salah-
Eddine Dahbi<EMAIL_ADDRESS>School of Physics and Institute for
Collider Particle Physics, University of the Witwatersrand, Johannesburg, Wits
2050, South Africa  Abhaya Kumar Swain<EMAIL_ADDRESS>School of
Physics and Institute for Collider Particle Physics, University of the
Witwatersrand, Johannesburg, Wits 2050, South Africa

###### Abstract

ATLAS observed a weaker than expected limit for di-jets resonances with a mass
slightly below $\approx$1 TeV. In addition, CMS reported hints for the (non-
resonant) pair production of di-jet resonances $X$ via a particle $Y$ at a
very similar mass range with a local (global) significance of 3.6 $\sigma$
(2.5 $\sigma$) at $m_{X}\approx 950\,$GeV. In this article we show that using
the preferred range for $m_{X}$ from the ATLAS analysis, one can reinterpret
the CMS analysis of di-di-jets in terms of a resonant search with $Y\to XX$,
with a significantly reduced look-elsewhere effect, finding an excess for
$m_{Y}\\!\approx\\!3.6$ TeVwith a significance of $4.4\sigma$ ($3.6\sigma$)
locally (globally). We present two possible UV completions capable of
explaining the (di-)di-jet excesses, one containing two scalar di-quarks, the
other one involving heavy gluons based on an
$SU(3)_{1}\\!\times\\!SU(3)_{2}\\!\times\\!SU(3)_{3}$ gauge symmetry,
spontaneously broken to $SU(3)$ color. In the latter case, non-perturbative
couplings are required, pointing towards a composite or extra-dimensional
framework. In fact, realizing such a model within the Randall-Sundrum space-
time (5D-AdS), one obtains the correct mass ratio for $m_{X}/m_{Y}$, assuming
the $X$ is the lowest lying resonance, and predicts a third (di-)di-jet
resonance with a mass around $\approx 2.2$ TeV.

††preprint: PSI-PR-22-27, ZU-TH 42/22

##  I Introduction

Since the discovery of the Higgs boson in 2012 [1, 2] the main focus of the
LHC has been on the discovery of new particles and new interactions beyond the
ones included in the Standard Model (SM) of particle physics. While intriguing
indirect signs emerged (see e.g. Refs. [3, 4, 5] for recent reviews), no new
resonance has been discovered yet. However, recently the number of hints for
new physics (NP) in direct LHC searches increased. In particular, ATLAS [6]
observed a weaker limit than expected in resonant di-jet searches111The
analogous CMS di-jet search [7] does not display an excess in the same region.
However, the sensitivity is significantly lower, such that the signal
suggested by the ATLAS analysis is not excluded. in a mass region slightly
below 1 TeV, while CMS [8] found hints for the (non-resonant) pair production
of di-jet resonances with a mass of $\approx 950$ GeV (see Fig. 2 in the
appendix) with a local (global) significance of 3.6 $\sigma$ (2.5 $\sigma$)
when integrating over the di-di-jet mass.

While the ATLAS analysis by itself does not constitute a significant hint for
beyond the SM physics once the look-elsewhere effect (LEE) is taken into
account, the compatibility of the suggested di-jet mass with the one of the
(non-resonant) CMS di-di-jet analysis is striking. This agreement suggests
that both excesses might be due to the same new particle $X$, once directly
(resonantly) produced in proton-proton collisions ($pp\\!\to\\!X\\!\to\\!jj$),
once pair produced via a new state $Y$
($pp\\!\to\\!Y^{(*)}\\!\to\\!XX\\!\to\\!(jj)(jj)$). While the CMS
collaboration in their analysis interprets the di-di-jet excess as the non-
resonant production of $XX$ (with $m_{X}\approx 950\,$GeV) via a heavy new
particle $Y$, with $m_{Y}\approx 8$ TeV, resulting in a local (global)
significance of $3.9\sigma$ ($1.6\sigma$) [8], it is also possible that the
two $X$ particles are resonantly produced from the decay of an on-shell $Y$
particle. In fact, the CMS results suggest 3
TeV$\lessapprox\\!m_{Y}\\!\lessapprox$4 TeV (see Fig. 3 in the appendix) for
such a resonant scenario, once $m_{X}$ is assumed to be within the preferred
range of the ATLAS di-jet analysis.

Figure 1: Feynman diagrams showing the resonant production of di-jets via the
particle $X$ (upper panel) and di-di-jets via the decay chain
$Y\\!\to\\!XX\\!\to\\!(jj)(jj)$ (lower panel). Note that $X$ and $Y$ could be
scalar or vector bosons in our setup. As we show in the main text,
$m_{X}\\!\approx\\!950\,$GeV and $m_{Y}\\!\approx\\!3.6\,$TeV is preferred by
the combination of the ATLAS and CMS analyses.

In order to evaluate this option more quantitatively, a (at least simplified)
model is necessary such that the experimental resolution and acceptance can be
simulated. We will do this in Sec. II using a simplified model with new vector
bosons in order to derive the significance resulting from the CMS analysis for
such a scenario with an on-shell $Y$ resonance decaying to two $X$ particles,
as illustrated in Fig 1. Next, we will examine possible UV completions that
can provide a common explanation of the (di-)di-jet excesses. As we will
discuss in Sec. III, two scalar di-quarks or new massive gluons seem to be the
most plausible candidates. Concerning the latter, we will consider a specific
example based on an $SU(3)_{1}\times SU(3)_{2}\times SU(3)_{3}$ gauge group,
broken down to $SU(3)$ color via two bi-triplets. We then conclude and present
an outlook in Sec. IV.

##  II (Di-)Di-Jets

As outlined in the introduction, the preferred value for the di-jet invariant
mass of ATLAS and CMS analyses strongly suggest that both signals are due to
the same particle $X$, i.e. that $pp\\!\to\\!X\to jj$ and
$pp\\!\to\\!Y\\!\to\\!XX\\!\to\\!(jj)(jj)$ account for the di-jet and the di-
di-jet excess, respectively (see Fig. 1). In this section we consider this
setup within a simplified model with a vector boson $Y$ decaying into two
vector bosons $X$.222In the next section we will consider models that could
provide a common explanation of the (di-)di-jet excesses. There we will also
consider a model with scalars. We did not explicitly simulate this setup,
however the differences compared to the case with gauge bosons is expected to
be small as the decay kinematics are very similar. We will assume that the
vectors have a $Y-X-X$ coupling, depending on the momenta in the same way as
the SM $Z-W-W$ coupling, with $m_{Y}>m_{X}$ and Br$[Y\to XX]=$100%. In
addition to this triple gauge boson interaction, only $X$ and $Y$ couplings to
SM quarks, which we assume to be flavour universal, are relevant.

First of all, we fix $900\,{\rm GeV}\lessapprox m_{X}\lessapprox 1050{\rm
GeV}$ from the invariant mass preferred by the di-jet analysis of ATLAS [6]
which is based on 29.3 fb-1 integrated luminosity at 13 TeV. Note that this
significantly reduces the LEE with respect to the di-jet invariant mass. We
then employ $m_{X}=950$ GeV, which corresponds to the best value obtained in
the non-resonant analysis by CMS. As such, we move on to the di-di-jet mass
$m_{Y}$ for which the CMS search for pairs of jets was performed with 139 fb-1
integrated luminosity at 13 TeV center of mass energy [8]. In this analysis,
four high transverse momentum jets are selected, including both the cases of
resonant $pp\\!\to\\!Y\\!\to\\!XX\\!\to\\!4j$ and non-resonant
$pp\\!\to\\!XX\\!\to\\!(jj)(jj)$ production. The observable

$\alpha=\frac{m_{1}+m_{2}}{2\cdot m_{4j}}\,,$ (1)

is defined, where $m_{1}$ and $m_{2}$ are the di-jet invariant masses and
$m_{4j}$ is the invariant mass of the four-jet system. The search is then
performed in bins of $\alpha$, and in the non-resonant case an excess at
$m_{Y}\approx 8.5\,$TeV with a local (global) significance of 3.9 $\sigma$
(1.6 $\sigma$) is reported. However, also a resonant-like excess in the four-
jet invariant mass spectrum around 3-4 TeV, i.e. for $\alpha=0.27,0.29,0.31$
with $m_{X}\approx 950\,$GeV, is visible. The cross-section of this four-jet
excess can naively be estimated to be of the order of $O$(fb).

In order to evaluate this possibility of a resonant production of $X(950)$
more quantitatively, we use our simplified model to simulate
$pp\\!\to\\!Y\\!\to\\!XX\to(jj)(jj)$ events using MadGraph5$\\_$aMC@NLO 2.6.7
with leading order (LO) accuracy in QCD [9]. The parton showering and
hadronization are simulated with PYTHIA 8.2 [10] using the NNPDF2.3 LO parton
distribution function set [11]. The events were processed with Delphes 3 [12],
which provides an approximate fast simulation of the detector. Jets were
reconstructed using the anti-$kt$ algorithm [13] with the radius parameter
$R=0.4$, as implemented in FastJet 3.2.2 [14]. Jets with $p_{\textrm{T}}>80$
GeV and $|\eta|<2.5$ are considered. Reconstructed jets overlapping with
photons, electrons or muons in a cone of size $R=0.4$ are then removed. The
four jets with the highest $p_{\textrm{T}}$ are considered as the leading
jets. Then the most probably di-jet pairs combination are created by
minimizing the $\eta-\phi$ space separations of the jets in each events:

$\Delta R=|(\Delta R_{1}-0.8)|+|(\Delta R_{2}-0.8)|\,,$ (2)

where $\Delta R_{1}$ and $\Delta R_{2}$ are the $\eta-\phi$333The distance
$\Delta R$ between two jets in the $\eta-\phi$ space is defined as $\Delta
R=\sqrt{(\Delta\eta_{jj})^{2}+(\Delta\phi_{jj})^{2}}$. space separations
between the two jets within the respective systems. The offset of 0.8 is
chosen to avoid the pairings with overlapped jets. In addition, we require the
$\Delta R_{i;i=1,2}$ to be less than 2, in order to reject contribution from
hard jets produced by QCD processes. While the pseudo-rapidity separation
$\Delta\eta_{jj}$ between the two jets of each di-jet systems is required to
be below 1.1, to remove contribution of backgrounds from QCD t-channel. In the
end, we required the asymmetry in the di-jet mass between the di-jet systems
to be small ($\frac{|m_{1}-m_{2}|}{m_{1}+m_{2}}<0.1$) which essentially select
the di-jets of equal mass taking into account the energy resolution. This, in
turn, is the property of a pair of equal mass resonances, which is unlike to
QCD jets that constitute the SM background.

The most significant signal is obtained for the bins with the central values
$\alpha=0.27$ and $\alpha=0.29$. Performing the weighted average of these
dominant bins, including the acceptance and resolution obtained by the
simulation, we find the best agreement with data for $m_{Y}\approx 3.6$ TeV
with a total cross-section for $pp\to Y\to XX\to jjjj$ of $\approx 5$fb. The
corresponding local (global) significance is $4.4\sigma$ ($3.6\sigma$). Note
that the global significance of our resonant excess is higher than the non-
resonant effect of CMS mainly due to the smaller LEE as we fixed the range of
the di-jet mass a priori with the help on the ATLAS data. The LEE effect
evaluated here includes the range $m_{Y}$ used in the search.

##  III Interpretation

###  III.1 Vector Bosons Based on $SU(3)^{3}$

A model with new vector bosons seems a natural possibility for providing a
common explanation of the di-di-jet excesses as such states can have sizable
couplings to valence quarks without breaking $SU(2)_{L}$ (similar to the SM
gauge bosons), and in fact already coupling of the order $10^{-1}$ turn out to
be sufficient to obtain suitable cross sections. Since self-interactions are
required to give rise to $Y\to XX$, this suggests that the new heavy vector
bosons originate from a non-abelian gauge group. Furthermore, if one wants
that all new vectors to couple to quarks, they must have the same quantum
numbers as the $SU(2)_{L}$ or $SU(3)_{c}$ gauge bosons of the SM, since
otherwise the operators $V_{\mu}^{a}\bar{q}\gamma^{\mu}T^{a}q$, where
$V_{\mu}^{a}$ are the new vector bosons and $T^{a}$ the corresponding
generators, would not be invariant under the SM gauge group. In addition, in
order to avoid couplings to leptons, which are strongly constrained from di-
lepton searches [15, 16], as well as bounds from electroweak precision
observables [17, 18], we will opt for a gauge group based on, and related to,
$SU(3)_{c}$.

Models with such additional heavy colored states based on an extended group
for the strong interactions, whose spontaneous symmetry breaking reduces it to
its diagonal subgroup, then identified with $SU(3)_{c}$, were proposed and
studied in Refs. [19, 20, 21, 22, 23, 24]. Furthermore, such a setup emerges
in the context of extra space-time dimensions where Kaluza-Klein excitations
of gluons exist [25, 26, 27, 28] and a similar picture arises in
composite/technicolor models [29, 30, 31].

As we need heavy resonances with (at least) two different masses, we consider
the gauge group

$SU(3)_{1}\times SU(3)_{2}\times SU(3)_{3}\times SU(2)_{L}\times U(1)_{Y}\,,$
(3)

broken down to the SM one $SU(3)_{c}\times SU(2)_{L}\times U(1)_{Y}$ via two
bi-doublet charged under two non-identical $SU(3)$ groups, each. Here we use

$\displaystyle\begin{tabular}[]{c|c c c c
c}&$SU(3)_{1}$&$SU(3)_{2}$&$SU(3)_{3}$\\\
\hline\cr$\Omega_{12}$&3&$\bar{3}$&1\\\
$\Omega_{23}$&1&3&$\bar{3}$\end{tabular}\qquad{\rm with}$ (7)
$\displaystyle\left\langle{{\Omega_{12}}}\right\rangle={v_{12}}\left({\begin{array}[]{*{20}{c}}1&0&0\\\
0&1&0\\\
0&0&1\end{array}}\right),\;\left\langle{{\Omega_{23}}}\right\rangle={v_{23}}\left({\begin{array}[]{*{20}{c}}1&0&0\\\
0&1&0\\\ 0&0&1\end{array}}\right),$ (14)

which constitutes a choice of basis, i.e. any other combination ($12,\,13$,
$13,\,23$) would lead to the same physical results. The spontaneous symmetry
breaking via the vacuum expectation values $v_{12}$ and $v_{23}$ leads to the
following mass matrix for the $SU(3)$ gauge fields $G^{\mu a}_{i}$ ($i=1,2,3$)
in the interaction basis

$L_{M}^{G}=\frac{1}{2}{\left({\begin{array}[]{*{20}{c}}{G_{1}^{\mu a}}\\\
{G_{2}^{\mu a}}\\\ {G_{3}^{\mu
a}}\end{array}}\right)^{T}}\left({\begin{array}[]{*{20}{c}}{v_{12}^{2}g_{1}^{2}}&{v_{12}^{2}g_{1}g_{2}}&0\\\
{v_{12}^{2}g_{1}g_{2}}&{v_{12}^{2}\left({g_{1}^{2}+g_{2}^{2}}\right)+v_{23}^{2}\left({g_{2}^{2}+g_{3}^{2}}\right)}&{v_{23}^{2}{g_{2}}{g_{3}}}\\\
0&{v_{23}^{2}{g_{2}}{g_{3}}}&{v_{23}^{2}\left({g_{2}^{2}+g_{3}^{2}}\right)}\end{array}}\right)\left({\begin{array}[]{*{20}{c}}{G_{1}^{\mu
a}}\\\ {G_{2}^{\mu a}}\\\ {G_{3}^{\mu a}}\end{array}}\right)\,,$ (15)

where each block corresponds to $a=1,...,8$ gauge bosons with the
corresponding generators $T^{a}$ and equal masses.

We can now diagonalize this mass matrix to obtain the mass eigenstates,
$g_{1}^{a\mu}$, $g_{2}^{a\mu}$ and $g_{3}^{a\mu}$ and identify the state with
the zero eigenvalue $g_{1}^{a\mu}$ with the SM gluons and the corresponding
coupling with the strong coupling constant $g_{s}$. The mass of $g_{2}^{a\mu}$
($g_{3}^{a\mu}$) should correspond to the $X$ ($Y$) resonance, i.e. $950\,$GeV
($3.6\,$TeV). We can furthermore determine the couplings of $g_{2}^{a\mu}$ and
$g_{3}^{a\mu}$ by demanding that the correct signal strength are obtained.
Since ATLAS finds a preferred value of $g_{q}\approx 0.07$ (in their
conventions where quarks couple only to the axial-vector current) for the $X$
resonance, and in our model we have 8 $g_{2}^{a\mu}$ fields which couples each
vectorially and flavour universal to quarks, we find that the production cross
section is 4 times larger (for equal couplings) resulting in
$g^{\prime}\approx 0.035$, where $g^{\prime}$ is the (effective) coupling of
$g_{2}^{a\mu}$ to SM quarks. The preferred value for the di-di-jet cross
section obtained in the last section is $\approx 5fb$. From this we find
$g^{\prime\prime}\approx 0.07/\sqrt{{\rm Br}[G^{\prime\prime}\to
G^{\prime}G^{\prime}]},$ by using the total production cross section for a
sequential SM $Z^{\prime}$ of this mass ($20\,$fb [32]) and taking into
account the $Z^{\prime}$ branching ratio and the PDF scaling, using the PDF of
Ref. [33] implemented in ManeParse [34], in order to rescale the cross section
to the one of our model.

We can now attempt to solve this system of equations if one specifies under
which $SU(3)_{i}$ gauge factors the SM quarks transform as a triplet. There
are seven possibilities for such charge assignments ($SU(3)_{1}$, $SU(3)_{2}$,
$SU(3)_{3}$, $SU(3)_{1}\\!\vee\\!SU(3)_{2}$, $SU(3)_{1}\\!\vee\\!SU(3)_{3}$,
$SU(3)_{1}\\!\vee\\!SU(3)_{3}$ and
$SU(3)_{1}\\!\vee\\!SU(2)_{1}\\!\vee\\!SU(3)_{3}$) among which only the option
that the SM quarks are $SU(3)_{1}$ triplets, but uncharged under both other
$SU(3)$ gauge factors, provides a solution. In fact, we find $g_{1}\approx 1$,
$g_{2}\approx 10$, $g_{3}\approx 15$ which is clearly in the non-perturbaitve
regime. This large values for the couplings $g_{2}$ and $g_{3}$ can be traced
back to the smallness of the $g_{2}^{a\mu}$ and $g_{3}^{a\mu}$ couplings to SM
quarks which requires small mixing among the colored gauge bosons.
Nonetheless, as the decay width to SM fermions is small and the right masses
and couplings can be obtained, this suggests that the gauge group
$SU(3)_{1}\times SU(3)_{2}\times SU(3)_{3}$, broken to $SU(3)_{c}$ via the
described breaking, can in fact explain the (di-)di-jet excesses. Furthermore,
the sizable couplings $g_{2}$ and $g_{3}$ point towards an extra-dimensional
or composite realization of this setup.

###  III.2 Scalar Di-Quarks

Alternatively to the vector-boson model proposed above, one could try to find
a perturbative explanation of the (di-)di-jet excesses using scalar bosons.
Because the suggested cross sections are too large to originate from a scalar
produced via gluon fusion (with perturbative couplings) [35], relevant
couplings to valence quarks are needed. Since $SU(3)_{c}$ singlet scalars can
only interact with quarks in the same way as the SM Higgs boson, the couplings
are naturally related to the respective Yukawa couplings, rendering them tiny
for valence quarks, thus resulting in too small cross sections.

However, $SU(3)_{c}$ triplet or sextuplet (symmetric $3\times 3$) scalars can
couple to quarks of the same $SU(2)_{L}$ representation such that their
couplings are unrelated to EW symmetry breaking and therefore also unrelated
to quark Yukawa couplings. Searches for such di-quarks via di-jet and di-di-
jet signatures were proposed in Refs. [36, 37, 38, 39, 40, 41, 42, 43].

The choice of quantum numbers for di-quarks is restricted to five
possibilities

| $SU(3)_{c}$ | $SU(2)_{L}$ | $U(1)_{Y}$
---|---|---|---
$\Phi_{u}$ | $\bar{6}$ | 1 | -4/3
$\Phi_{d}$ | $\bar{6}$ | 1 | 2/3
$\Phi_{q}^{(1)}$ | 3 | 1 | -1/3
$\Phi_{q}^{(3)}$ | $\bar{6}$ | 3 | -1/3
$\Phi_{ud}$ | $\bar{6}$ | 1 | 1
(16)

if we restrict our selves to the cases which allow couplings symmetric in
flavour space. Note that anti-symmetric couplings would in general cause
problems with $\Delta F=2$ processes. In this case we have the coupling to
fermions

$\displaystyle\mathcal{L}_{\rm int}=$
$\displaystyle\lambda_{u}\bar{u}_{R}^{c}\Phi_{u}u_{R}+\lambda_{d}\bar{d}_{R}^{c}\Phi_{d}d_{R}+\lambda_{q}^{(3)}\Phi_{q}^{(3)I}\,\bar{q}_{L}^{c}i\sigma_{2}\,\tau^{I}q_{L}$
$\displaystyle+\lambda_{q}^{(1)}\epsilon\Phi_{q}^{(1)}\,\bar{q}_{L}^{c}i\sigma_{2}\,q_{L}+\lambda_{ud}\bar{u}_{R}^{c}\Phi_{ud}d_{R}+{\rm
h.c.}\,,$ (17)

where $u_{R},\,d_{R}$ and $q_{L}$ are the SM right-handed $SU(2)_{L}$ singlet
quarks and left-handed $SU(2)_{L}$ doublet quarks, respectively. $\epsilon$ is
the totally anti-symmetric tensor in three dimensions which contracts the
implicit color indices of the color triplets and we suppressed flavour
indices.

We can thus attempt to construct a scenarios which has the potential to
reproduce the experimental signals. Assuming that $\Phi_{u}$ is the $3.6\,$TeV
resonance decaying into two $\Phi_{d}$ scalars with a mass of $950\,$ GeV
each, the interaction term $\xi\,\epsilon\epsilon\Phi_{u}\Phi_{d}\Phi_{d}$ is
needed where the first (second) $\epsilon$ contracts the first (second)
$SU(3)_{c}$ index of the symmetric $3\times 3$ representations. Since both
scalars couple to right-handed $SU(2)_{L}$ singlet quarks, we can assume that
they have flavour diagonal couplings, both in the interaction and in the mass
basis. However, as the couplings to first generation quarks are constrained by
neutron-anti-neutron oscillations [44], one has to assume that the couplings
to second generation quarks are dominant (at least for either $\Phi_{u}$ or
$\Phi_{d}$). These couplings are then determined by requiring that the correct
signal strengths are obtained.444Note that neither the CMS nor the ATLAS
analysis is sensitive to the electric charge of the vector because the jet
charge is not measured. In addition, the differences in efficiencies between
scalar and vector resonances are expected to be small for the analyses under
investigation. The ATLAS di-jet analysis gives $|\lambda_{d(s)}|\simeq
0.05(0.2)$. In addition, assuming ${\rm Br}(\Phi_{u}\to\Phi_{d}\Phi_{d})\simeq
100\%$, which is natural for $\xi=O(1)$), we find $|\lambda_{u(c)}|\simeq
0.02(1.1)$ from the di-di-jet cross section of $\approx 5\,$fb.

In principle, also the option that $\Phi_{d}$ is the $3.6\,$TeV resonance and
$\Phi_{q}^{(3)}$ or $\Phi_{q}^{(1)}$ the $950\,$GeV one is possible. In this
case, it has to be assumed that the couplings to quarks are universal, such
that the CKM rotation between the interaction and the mass eigenbasis does not
generate flavour changing couplings that would contribute to $\Delta F=2$
processes.

##  IV Conclusions and Outlook

In this article we pointed out that the ATLAS di-jet excess with a mass
slightly below 1 TeV is perfectly consistent with the preferred di-jet mass of
$950\,$GeV of the CMS di-di-jet analysis. We then used the suggested range for
$m_{X}$ from ATLAS to recast the CMS di-di-jet analysis in terms of a resonant
search for $Y\\!\to\\!XX\\!\to\\!(jj)(jj)$. This significantly reduces the LEE
and results in a local (global) significance of $4.4\,\sigma$ $(3.6\,\sigma)$
for a resonance $Y$ with mass $m_{Y}\approx 3.6\,\text{TeV}$.

We then examined possible combined explanations of the (di-)di-jet excesses
and proposed both a model with scalar di-quarks and a model with new heavy
colored vector bosons based on an
$SU(3)_{1}\\!\times\\!SU(3)_{2}\\!\times\\!SU(3)_{3}$ gauge symmetry
spontaneously broken to $SU(3)_{c}$. While the scalar di-quark model has
couplings that are at most the order one, the $SU(3)^{3}$ model requires large
non-perturbative couplings, pointing towards an extra-dimensional or composite
realization. Interestingly, interpreting this model in a Randall-Sundrum (RS)
framework [45], the ratio of the masses of the gauge boson excitations are
predicted to be [46]

$m_{n}/m_{1}=4(n-1/4)/3\,,$ (18)

where $m_{1}$ is the first gluon excitation with a non-vanishing mass. This
means if the first resonance ($n=1$) is at $\approx 950\,$GeV, the second one
($n=2$) should be at $\approx\\!2.2\,$TeV while the third ($n=2$) is at
$\approx 3.5\,$TeV. While the latter value fits nicely the (di-)di-jet data,
this RS framework predicts the existence of another (di-)di-jet resonance with
a mass around $2.2\,$TeV. Note that such a resonance, if it has similar
couplings to quarks as the $n=1$ and $n=3$ resonances, is not excluded by
current di-jet searches due to the PDF scaling w.r.t. the $950\,$GeV
resonance. Furthermore, the CMS di-di-jet data even points towards a slight
excess in this region of the di-di-jet invariant mass $m_{Y}$ (see bottom-left
panel ($m_{X}/m_{Y}=0.17$) in Fig. 3).

In light of the intriguing hints for NP in semi-leptonic $B$ decays [47, 48],
$g-2$ of the muon [49, 50, 51], the $W$ mass [52, 53], the Cabibbo angle
anomaly [54, 55, 56], the $96\,\sigma$GeV [57], $151\,\sigma$GeV [58] and
$680\,$GeV [59] excesses, the multi-lepton anomalies [60, 61, 62, 63], the di-
Higgs [64] excess as well as the hint for non-resonant di-electrons [16,
65],555See Ref. [5] for a recent review of anomalies. the (di-)di-jet excesses
constitute one more very interesting sign of physics beyond the SM. While the
other signals for NP are in general related electroweak processes within the
SM, the (di-)di-jet signals points towards colored new particles. This
broadens the range of interactions for which the anomalies suggest NP and has
important consequences for collider searches and model building in the
collaborative search for the next SM of particle physics.

Acknowledgements — We thank Zurab Berezhiani, Monika Blanke, Mukesh Kumar and
Xifeng Ruan for useful discussions. B.M. gratefully acknowledges the support
from the South African Department of Science and Innovation through the SA-
CERN program and the National Research Foundation for various forms of
support. A.C. is supported by a Professorship Grant (PP00P2_176884) of the
Swiss National Science Foundation.

##  Appendix A Plots from the CMS and ATLAS Analyses

Figure 2: Left: Di-jet search of ATLAS [6] showing the expected and observed
limits on the axial coupling $g_{q}$ of a $Z^{\prime}$ boson to quarks. Right:
Cross section times branching ratio times acceptance in units of pico barn as
a function of the di-jet invariant mass obtained in the CMS di-di-jet analysis
[8].

Here we quote the main results of the ATLAS and CMS searches for (di-)di-jet
searches for the convenience of the reader. The result of the di-jet resonance
search of ATLAS is shown in the left plot of Fig. 2. The di-jet invariant mass
$m_{X}$ of the CMS di-di-jet analysis is given in the right plot of Fig. 2
while the relevant plots for the di-di-jet mass $m_{Y}$ are displayed in Fig.
3.

Figure 3: Observed and expected limit on cross section times branching ratio
times acceptance in units of pico barn as a function of the di-di-jet
invariant mass for different values of $\alpha=m_{X}/m_{Y}$ [8].

## References

  * Aad  _et al._ [2012] G. Aad _et al._ (ATLAS), Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC, Phys. Lett. B 716, 1 (2012), arXiv:1207.7214 [hep-ex] .
  * Chatrchyan  _et al._ [2012] S. Chatrchyan _et al._ (CMS), Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC, Phys. Lett. B 716, 30 (2012), arXiv:1207.7235 [hep-ex] .
  * Crivellin and Matias [2022] A. Crivellin and J. Matias, Beyond the Standard Model with Lepton Flavor Universality Violation, in  _$1^{\rm st}$ Pan-African Astro-Particle and Collider Workshop_ (2022) arXiv:2204.12175 [hep-ph] .
  * Crivellin and Hoferichter [2021] A. Crivellin and M. Hoferichter, Hints of lepton flavor universality violations, Science 374, 1051 (2021), arXiv:2111.12739 [hep-ph] .
  * Fischer  _et al._ [2022] O. Fischer _et al._ , Unveiling hidden physics at the LHC, Eur. Phys. J. C 82, 665 (2022), arXiv:2109.06065 [hep-ph] .
  * Aaboud  _et al._ [2018] M. Aaboud _et al._ (ATLAS), Search for low-mass dijet resonances using trigger-level jets with the ATLAS detector in $pp$ collisions at $\sqrt{s}=13$ TeV, Phys. Rev. Lett. 121, 081801 (2018), arXiv:1804.03496 [hep-ex] .
  * Sirunyan  _et al._ [2018] A. M. Sirunyan _et al._ (CMS), Search for narrow and broad dijet resonances in proton-proton collisions at $\sqrt{s}=13$ TeV and constraints on dark matter mediators and other new particles, JHEP 08, 130, arXiv:1806.00843 [hep-ex] .
  * CMS [2022] Search for resonant and nonresonant production of pairs of dijet resonances in proton-proton collisions at $\sqrt{s}$ = 13 TeV (2022), arXiv:2206.09997 [hep-ex] .
  * Alwall  _et al._ [2014] J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations, JHEP 07, 079, arXiv:1405.0301 [hep-ph] .
  * Sjöstrand  _et al._ [2015] T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, An introduction to PYTHIA 8.2, Comput. Phys. Commun. 191, 159 (2015), arXiv:1410.3012 [hep-ph] .
  * Ball  _et al._ [2013a] R. D. Ball _et al._ , Parton distributions with LHC data, Nucl. Phys. B 867, 244 (2013a), arXiv:1207.1303 [hep-ph] .
  * de Favereau  _et al._ [2014] J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi (DELPHES 3), DELPHES 3, A modular framework for fast simulation of a generic collider experiment, JHEP 02, 057, arXiv:1307.6346 [hep-ex] .
  * Cacciari  _et al._ [2008] M. Cacciari, G. P. Salam, and G. Soyez, The anti-$k_{t}$ jet clustering algorithm, JHEP 04, 063, arXiv:0802.1189 [hep-ph] .
  * Cacciari  _et al._ [2012] M. Cacciari, G. P. Salam, and G. Soyez, FastJet User Manual, Eur. Phys. J. C 72, 1896 (2012), arXiv:1111.6097 [hep-ph] .
  * Aad  _et al._ [2020] G. Aad _et al._ (ATLAS), Search for new non-resonant phenomena in high-mass dilepton final states with the ATLAS detector, JHEP 11, 005, [Erratum: JHEP 04, 142 (2021)], arXiv:2006.12946 [hep-ex] .
  * Sirunyan  _et al._ [2021] A. M. Sirunyan _et al._ (CMS), Search for resonant and nonresonant new phenomena in high-mass dilepton final states at $\sqrt{s}$ = 13 TeV, JHEP 07, 208, arXiv:2103.02708 [hep-ex] .
  * Schael  _et al._ [2006] S. Schael _et al._ (ALEPH, DELPHI, L3, OPAL, SLD, LEP Electroweak Working Group, SLD Electroweak Group, SLD Heavy Flavour Group), Precision electroweak measurements on the $Z$ resonance, Phys. Rept. 427, 257 (2006), arXiv:hep-ex/0509008 .
  * Schael  _et al._ [2013] S. Schael _et al._ (ALEPH, DELPHI, L3, OPAL, LEP Electroweak), Electroweak Measurements in Electron-Positron Collisions at W-Boson-Pair Energies at LEP, Phys. Rept. 532, 119 (2013), arXiv:1302.3415 [hep-ex] .
  * Hill [1991] C. T. Hill, Topcolor: Top quark condensation in a gauge extension of the standard model, Phys. Lett. B 266, 419 (1991).
  * Frampton and Glashow [1987] P. H. Frampton and S. L. Glashow, Chiral Color: An Alternative to the Standard Model, Phys. Lett. B 190, 157 (1987).
  * Chivukula  _et al._ [1996] R. S. Chivukula, A. G. Cohen, and E. H. Simmons, New strong interactions at the Tevatron?, Phys. Lett. B 380, 92 (1996), arXiv:hep-ph/9603311 .
  * Martynov and Smirnov [2009] M. V. Martynov and A. D. Smirnov, Chiral color symmetry and possible G-prime-boson effects at the Tevatron and LHC, Mod. Phys. Lett. A 24, 1897 (2009), arXiv:0906.4525 [hep-ph] .
  * Frampton  _et al._ [2010] P. H. Frampton, J. Shu, and K. Wang, Axigluon as Possible Explanation for p anti-p —$>$ t anti-t Forward-Backward Asymmetry, Phys. Lett. B 683, 294 (2010), arXiv:0911.2955 [hep-ph] .
  * Chivukula  _et al._ [2013] R. S. Chivukula, E. H. Simmons, and N. Vignaroli, A Flavorful Top-Coloron Model, Phys. Rev. D 87, 075002 (2013), arXiv:1302.1069 [hep-ph] .
  * Han  _et al._ [1999] T. Han, J. D. Lykken, and R.-J. Zhang, On Kaluza-Klein states from large extra dimensions, Phys. Rev. D 59, 105006 (1999), arXiv:hep-ph/9811350 .
  * Dicus  _et al._ [2002] D. A. Dicus, C. D. McMullen, and S. Nandi, Collider implications of Kaluza-Klein excitations of the gluons, Phys. Rev. D 65, 076007 (2002), arXiv:hep-ph/0012259 .
  * Davoudiasl  _et al._ [2001] H. Davoudiasl, J. L. Hewett, and T. G. Rizzo, Experimental probes of localized gravity: On and off the wall, Phys. Rev. D 63, 075004 (2001), arXiv:hep-ph/0006041 .
  * Lillie  _et al._ [2007] B. Lillie, L. Randall, and L.-T. Wang, The Bulk RS KK-gluon at the LHC, JHEP 09, 074, arXiv:hep-ph/0701166 .
  * Dimopoulos [1980] S. Dimopoulos, Technicolored Signatures, Nucl. Phys. B 168, 69 (1980).
  * Farhi and Susskind [1981] E. Farhi and L. Susskind, Technicolor, Phys. Rept. 74, 277 (1981).
  * Hill  _et al._ [1993] C. T. Hill, D. C. Kennedy, T. Onogi, and H.-L. Yu, Spontaneously broken technicolor and the dynamics of virtual vector technimesons, Phys. Rev. D 47, 2940 (1993), arXiv:hep-ph/9210233 .
  * ATL [2019] Search for high-mass dilepton resonances using $139\,\mathrm{fb}^{-1}$ of $pp$ collision data collected at $\sqrt{s}=13\,\mathrm{TeV}$ with the ATLAS detector (2019).
  * Ball  _et al._ [2013b] R. D. Ball, V. Bertone, S. Carrazza, L. Del Debbio, S. Forte, A. Guffanti, N. P. Hartland, and J. Rojo (NNPDF), Parton distributions with QED corrections, Nucl. Phys. B 877, 290 (2013b), arXiv:1308.0598 [hep-ph] .
  * Clark  _et al._ [2017] D. B. Clark, E. Godat, and F. I. Olness, ManeParse: A Mathematica reader for Parton Distribution Functions, Comput. Phys. Commun. 216, 126 (2017), arXiv:1605.08012 [hep-ph] .
  * de Florian  _et al._ [2016] D. de Florian _et al._ (LHC Higgs Cross Section Working Group), Handbook of LHC Higgs Cross Sections: 4. Deciphering the Nature of the Higgs Sector (2016), arXiv:1610.07922 [hep-ph] .
  * Cakir and Sahin [2005] O. Cakir and M. Sahin, Resonant production of diquarks at high energy $pp$, ep and $e^{+}e^{-}$ colliders, Phys. Rev. D 72, 115011 (2005), arXiv:hep-ph/0508205 .
  * Mohapatra  _et al._ [2008] R. N. Mohapatra, N. Okada, and H.-B. Yu, Diquark Higgs at LHC, Phys. Rev. D 77, 011701 (2008), arXiv:0709.1486 [hep-ph] .
  * Chen  _et al._ [2009] C.-R. Chen, W. Klemm, V. Rentala, and K. Wang, Color Sextet Scalars at the CERN Large Hadron Collider, Phys. Rev. D 79, 054002 (2009), arXiv:0811.2105 [hep-ph] .
  * Kilic  _et al._ [2008] C. Kilic, T. Okui, and R. Sundrum, Colored Resonances at the Tevatron: Phenomenology and Discovery Potential in Multijets, JHEP 07, 038, arXiv:0802.2568 [hep-ph] .
  * Berger  _et al._ [2010] E. L. Berger, Q.-H. Cao, C.-R. Chen, G. Shaughnessy, and H. Zhang, Color Sextet Scalars in Early LHC Experiments, Phys. Rev. Lett. 105, 181802 (2010), arXiv:1005.2622 [hep-ph] .
  * Bai and Shelton [2012] Y. Bai and J. Shelton, Composite Octet Searches with Jet Substructure, JHEP 07, 067, arXiv:1107.3563 [hep-ph] .
  * Dobrescu  _et al._ [2018] B. A. Dobrescu, R. M. Harris, and J. Isaacson, Ultraheavy resonances at the LHC: beyond the QCD background (2018), arXiv:1810.09429 [hep-ph] .
  * Pascual-Dias  _et al._ [2020] B. Pascual-Dias, P. Saha, and D. London, LHC Constraints on Scalar Diquarks, JHEP 07, 144, arXiv:2006.13385 [hep-ph] .
  * Baldo-Ceolin  _et al._ [1994] M. Baldo-Ceolin _et al._ , A New experimental limit on neutron - anti-neutron oscillations, Z. Phys. C 63, 409 (1994).
  * Randall and Sundrum [1999] L. Randall and R. Sundrum, A Large mass hierarchy from a small extra dimension, Phys. Rev. Lett. 83, 3370 (1999), arXiv:hep-ph/9905221 .
  * Gherghetta and Pomarol [2000] T. Gherghetta and A. Pomarol, Bulk fields and supersymmetry in a slice of AdS, Nucl. Phys. B 586, 141 (2000), arXiv:hep-ph/0003129 .
  * Aaij  _et al._ [2022] R. Aaij _et al._ (LHCb), Test of lepton universality in beauty-quark decays, Nature Phys. 18, 277 (2022), arXiv:2103.11769 [hep-ex] .
  * Amhis  _et al._ [2021] Y. S. Amhis _et al._ (HFLAV), Averages of b-hadron, c-hadron, and $\tau$-lepton properties as of 2018, Eur. Phys. J. C 81, 226 (2021), arXiv:1909.12524 [hep-ex] .
  * Bennett  _et al._ [2006] G. W. Bennett _et al._ (Muon g-2), Final Report of the Muon E821 Anomalous Magnetic Moment Measurement at BNL, Phys. Rev. D 73, 072003 (2006), arXiv:hep-ex/0602035 .
  * Abi  _et al._ [2021] B. Abi _et al._ (Muon g-2), Measurement of the Positive Muon Anomalous Magnetic Moment to 0.46 ppm, Phys. Rev. Lett. 126, 141801 (2021), arXiv:2104.03281 [hep-ex] .
  * Aoyama  _et al._ [2019] T. Aoyama, T. Kinoshita, and M. Nio, Theory of the Anomalous Magnetic Moment of the Electron, Atoms 7, 28 (2019).
  * Aaltonen  _et al._ [2022] T. Aaltonen _et al._ (CDF), High-precision measurement of the W boson mass with the CDF II detector, Science 376, 170 (2022).
  * Zyla  _et al._ [2020] P. A. Zyla _et al._ (Particle Data Group), Review of Particle Physics, PTEP 2020, 083C01 (2020).
  * Seng [2022] C.-Y. Seng, First row CKM unitarity, in  _20th Conference on Flavor Physics and CP Violation_ (2022) arXiv:2207.10492 [hep-ph] .
  * Manzari [2022] C. A. Manzari, Explaining the Cabibbo Angle Anomaly, PoS EPS-HEP2021, 526 (2022), arXiv:2111.04519 [hep-ph] .
  * Crivellin [2022] A. Crivellin, Explaining the Cabibbo Angle Anomaly (2022) arXiv:2207.02507 [hep-ph] .
  * CMS [2017] Search for new resonances in the diphoton final state in the mass range between 70 and 110 GeV in pp collisions at $\sqrt{s}=$ 8 and 13 TeV (2017).
  * Crivellin  _et al._ [2021a] A. Crivellin, Y. Fang, O. Fischer, A. Kumar, M. Kumar, E. Malwa, B. Mellado, N. Rapheeha, X. Ruan, and Q. Sha, Accumulating Evidence for the Associate Production of a Neutral Scalar with Mass around 151 GeV (2021a), arXiv:2109.02650 [hep-ph] .
  * Aaboud  _et al._ [2017] M. Aaboud _et al._ (ATLAS), Search for new phenomena in high-mass diphoton final states using 37 fb-1 of proton–proton collisions collected at $\sqrt{s}=13$ TeV with the ATLAS detector, Phys. Lett. B 775, 105 (2017), arXiv:1707.04147 [hep-ex] .
  * von Buddenbrock  _et al._ [2016] S. von Buddenbrock, N. Chakrabarty, A. S. Cornell, D. Kar, M. Kumar, T. Mandal, B. Mellado, B. Mukhopadhyaya, R. G. Reed, and X. Ruan, Phenomenological signatures of additional scalar bosons at the LHC, Eur. Phys. J. C 76, 580 (2016), arXiv:1606.01674 [hep-ph] .
  * Buddenbrock  _et al._ [2019] S. Buddenbrock, A. S. Cornell, Y. Fang, A. Fadol Mohammed, M. Kumar, B. Mellado, and K. G. Tomiwa, The emergence of multi-lepton anomalies at the LHC and their compatibility with new physics at the EW scale, JHEP 10, 157, arXiv:1901.05300 [hep-ph] .
  * von Buddenbrock  _et al._ [2020] S. von Buddenbrock, R. Ruiz, and B. Mellado, Anatomy of inclusive $t\bar{t}W$ production at hadron colliders, Phys. Lett. B 811, 135964 (2020), arXiv:2009.00032 [hep-ph] .
  * Hernandez  _et al._ [2021] Y. Hernandez, M. Kumar, A. S. Cornell, S.-E. Dahbi, Y. Fang, B. Lieberman, B. Mellado, K. Monnakgotla, X. Ruan, and S. Xin, The anomalous production of multi-lepton and its impact on the measurement of $Wh$ production at the LHC, Eur. Phys. J. C 81, 365 (2021), arXiv:1912.00699 [hep-ph] .
  * ATL [2021] Combination of searches for non-resonant and resonant Higgs boson pair production in the $b\bar{b}\gamma\gamma$, $b\bar{b}\tau^{+}\tau^{-}$ and $b\bar{b}b\bar{b}$ decay channels using $pp$ collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector (2021).
  * Crivellin  _et al._ [2021b] A. Crivellin, C. A. Manzari, and M. Montull, Correlating nonresonant di-electron searches at the LHC to the Cabibbo-angle anomaly and lepton flavor universality violation, Phys. Rev. D 104, 115016 (2021b), arXiv:2103.12003 [hep-ph] .