Figure 25: Event distribution of FC $\nu_{e}$ CC candidates as a function reconstructed $E_{\nu}$. Same set-up as that of Fig. 24 except that the constraint from PC $\nu_{e}$ CC, FC $\nu_{\mu}$ CC, PC $\nu_{\mu}$ CC, FC CC$\pi^{0}$, PC CC$\pi^{0}$, and NC$\pi^{0}$ is used.  Category | w/o constraint | w/ constraint
---|---|---
Beam $\nu_{e}$CC | 42.6 $\pm$ 10.6 | 51.5 $\pm$ 2.6
$\nu_{\mu}$CC $\pi^{0}$ | 0.6 $\pm$ 0.8 | 0.8 $\pm$ 0.8
$\nu_{\mu}$CC (non-$\pi^{0}$) | 3.9 $\pm$ 4.2 | 3.1 $\pm$ 3.1
NC $\pi^{0}$ | 4.5 $\pm$ 2.3 | 4.3 $\pm$ 1.6
NC (non-$\pi^{0}$) | 3.0 $\pm$ 1.4 | 2.9 $\pm$ 1.2
Out of FV | 3.8 $\pm$ 2.0 | 3.4 $\pm$ 1.6
Dirt | 1.0 $\pm$ 1.0 | 1.2 $\pm$ 0.9
Cosmic | 0.3 $\pm$ 0.6 | 0.5 $\pm$ 0.6
EXT (beam-off data) | 1.9 $\pm$ 1.7
Predicted total (eLEEx=0) | 61.5 $\pm$ 15.3 $\pm$ 7.7 | 69.6 $\pm$ 5.0 $\pm$ 8.0
Predicted total (eLEEx=1) | 91.8 $\pm$ 23.4 $\pm$ 8.7 | 103.8 $\pm$ 7.4 $\pm$ 9.0
BNB data | 56
Table 4: Expected number of events for each type of background, beam intrinsic
$\nu_{e}$ CC, and eLEEx=1 $\nu_{e}$ CC in the energy region of reconstructed
$E_{\nu}<$600 MeV for FC $\nu_{e}$ CC, without and with all available
constraints. All the expected numbers correspond to 6.37$\times$1020 POT. The
uncertainties are systematic uncertainties for each category of the background
and $\nu_{e}$ CC signal except for the predicted total where the second
uncertainty is the statistical uncertainty using the CNP formalism. See text
in Sec. VII.1 for more detailed discussion.

The data comparisons to the energy spectra of the MC predictions before and
after constraint, without and with eLEEx=1 $\nu_{e}$ CC, for the FC $\nu_{e}$
CC channel are shown in Fig. 26. No significant discrepancy between data and
the MC eLEEx=0 hypothesis (obtained from the BNB beam simulation with the
MicroBooNE tuned Genie interaction models) was observed after applying all
available constraints. The two data points between 500 MeV and 700 MeV show
deficits compared to the constrained prediction of the eLEEx=0 hypothesis. A
quantitative examination considering full uncertainties yields a local p-value
of 0.039 (2.1$\sigma$), which is consistent with the hypothesis of statistical
fluctuation and/or systematic variations. The goodness-of-fit $\chi^{2}/ndf$
(Pearson format) and $p$-values for the entire energy region as well as the
low energy region (less than 600 MeV) are summarized in Table 5. Comparing the
$\chi^{2}/ndf$ values, one can see the impact from the constraint on the
sensitivity to the eLEE search as well as the level at which the current data
measurement excludes the eLEEx=1 hypothesis. More detailed statistical
analyses are reported in the following sections.

MicroBooNE 6.369$\times$1020 POTFC

(a) eLEEx=0 hypothesis.

MicroBooNE 6.369$\times$1020 POTFC

(b) eLEEx=1 hypothesis

Figure 26: Event distributions of FC $\nu_{e}$ CC candidates as a function of
reconstructed $E_{\nu}$. (a) An eLEEx=0 hypothesis and (b) an eLEEx=1
hypothesis are assumed in MC prediction and uncertainty calculation,
respectively. Black points are from data measurements. Red (blue) histograms
and error bands are for MC prediction before (after) constraint from the other
six channels. $\chi^{2}/ndf$, eLEEx=0
---
Energy region |  w/o constraint |  w/ constraint
(0, 2500) MeV |  12.55/25 $p_{\text{val}}=0.982$ |  17.86/25 $p_{\text{val}}=0.848$
(0, 600) MeV |  4.25/6 $p_{\text{val}}=0.643$ |  5.78/6 $p_{\text{val}}=0.448$
$\chi^{2}/ndf$, eLEEx=1
Energy region |  w/o constraint |  w/ constraint
(0, 2500) MeV |  13.02/25 $p_{\text{val}}=0.976$ |  28.24/25 $p_{\text{val}}=0.297$
(0, 600) MeV |  4.23/6 $p_{\text{val}}=0.646$ |  15.73/6 $p_{\text{val}}=0.015$
Table 5: Summary of the goodness-of-fit $\chi^{2}/ndf$ values of FC
$\nu_{e}$CC energy distributions without or with the constraint from the other
six channels for eLEEx=0 and eLEEx=1 hypotheses, respectively. In all model
goodness-of-fit tests, Pearson construction of $\chi^{2}$, instead of the CNP
construction of $\chi^{2}$, is used. The $p$-value for each goodness-of-fit
test is also shown.

###  VII.2 Simple vs. Simple Likelihood Ratio Test

Using the $\chi^{2}$ (CNP format) as defined in Eq. (7), we construct a
simple-vs-simple test statistic:

$\Delta\chi_{\text{simple}}^{2}=\chi^{2}|_{\text{eLEEx=1}}-\chi^{2}|_{\text{eLEEx=0}}$
(13)

where $x$ represents the expected eLEE strength in the prediction. This test
statistic allows one to calculate the p-value using a frequentist approach
(i.e. with pseudo experiments) assuming the MC prediction with the eLEEx=0
hypothesis or with the eLEEx=1 hypothesis is true. In generating the pseudo
experiments, the events in all channels are randomized according to their
associated systematic uncertainties (covariance matrix) and statistical
uncertainties (Poisson distribution). In the case the expected counts in a
given bin is negative, which can happen for large systematic uncertainties,
the corresponding pseudo experiment is discarded and a new psuedo experiment
is generated. This test statistic tests the compatibility between the data
from all seven channels and the given hypothesis. Note that as opposed to
Table 5 which focuses on the low energy FC $\nu_{e}$ CC events, both the FC
$\nu_{e}$ CC channel and the PC $\nu_{e}$ CC channel that have eLEE
sensitivity and the other $\nu_{\mu}$ CC and $\pi^{0}$ channels are
simultaneously employed in the hypothesis test here and in the following
section, in which case the $\nu_{\mu}$ CC and $\pi^{0}$ channels effectively
serve as constraints.

MicroBooNE 6.369$\times$1020 POT

(a) eLEEx=0 hypothesis.

MicroBooNE 6.369$\times$1020 POT

(b) eLEEx=1 hypothesis.

Figure 27: Distributions of simple-vs-simple $\Delta\chi^{2}_{\text{simple}}$
assuming (a) the eLEEx=0 hypothesis is true and (b) the eLEEx=1 hypothesis is
true. Both distributions are obtained from 4 million pseudo experiments. Data
$\Delta\chi^{2}_{\text{simple}}$ values are indicated by dashed vertical
lines.

Figure 27 shows the $\Delta\chi^{2}_{\text{simple}}$ values and distributions
of data and pseudo experiments for eLEEx=0 and eLEEx=1 hypotheses separately.
Assuming the eLEEx=0 hypothesis is true, the p-value of the data measurement
is derived to be 0.326 (one-sided), which corresponds to a 0.45$\sigma$
significance level. This result demonstrates good agreement between the data
measurement and the eLEEx=0 hypothesis. Assuming the eLEEx=1 hypothesis is
true, the p-value of the data measurement is derived to be $9.0\times 10^{-5}$
(one-sided), which disfavors the eLEEx=1 hypothesis at a 3.75$\sigma$
significance level.

###  VII.3 Nested Likelihood Ratio Test and Best-fit of eLEE strength

Besides the simple-vs-simple test statistic, another test statistic is
constructed to further test the compatibility between observation in all seven
channels and various hypotheses and to find the best fit of an eLEE model
where the strength of the eLEE signal, $x$, is allowed to vary such that $x=1$
would be the eLEE model strength based on the extrapolation from MiniBooNE and
values less than or greater than that indicate a smaller or larger excess
compared to MiniBooNE. The test statistic is defined as follows:

$\Delta\chi_{\text{nested}}^{2}=\chi^{2}|_{\text{eLEEx=x}_{0}}-\chi_{\text{min}}^{2}|_{\text{eLEEx=x}_{\text{min}}},~{}~{}x_{\text{min}}\geq
0.$ (14)

The value of $x_{0}$ represents the null hypothesis: $x_{0}=0$ represents the
eLEEx=0 hypothesis (expectation from the beam simulation and the tuned Genie
interaction models), and $x_{0}=1$ represents the eLEEx=1 hypothesis.
$x_{\text{min}}$ is the best-fit value of $x$ in the allowed region
($x_{\text{min}}\geq 0$) after minimization. This test statistic is a nested
likelihood ratio test statistic comparing the null hypothesis ($x=x_{0}$) with
an alternative hypothesis corresponding to $x_{min}$, which can in principle
take any value within the allowed region, and is included to quantify the
search for an eLEE along with the Feldman-Cousins approach [55].

The best-fit value of the eLEE strength $x_{min}$ is determined by minimizing
$\chi^{2}_{\text{eLEEx}}$ with the covariance matrix as defined in Eq. (8)
that accounts for the statistical and systematic uncertainties varying with
different eLEE strengths. As shown in Fig. 28, the best-fit eLEE strength is
at the boundary $x_{\text{min}}=0$ which corresponds to the eLEEx=0
hypothesis. The Feldman-Cousins 68.3%, 95.5% and 99.7% confidence level upper
limits from the data measurement are calculated to be 0.217, 0.513, and 0.911,
with the expected upper limits for the eLEEx=0 hypothesis at 0.243, 0.563, and
0.958, respectively.

In comparison, the 68% confidence interval (CI) of the eLEE strength estimated
from the MiniBooNE 2020 result [21] with full and statistical-only
uncertainties are shown in Fig. 28 as well. The lower limits of the 68%
MiniBooNE full and statistical-only CIs are disfavored at significance levels
of more than 2.6$\sigma$ and 3.0$\sigma$, respectively. Note that some of the
systematic uncertainties are correlated between the MiniBooNE and MicroBooNE
experiments and a direct comparison of the two experiments taking into account
correlated systematic uncertainties is unavailable. Therefore two separate
significance levels derived based on the published information are used to
indicate the bounds of this comparison.

MicroBooNE 6.369$\times$1020 POT

Figure 28: The $\Delta\chi^{2}_{\text{nested}}$ with the nested likelihood
ratio test. The best-fit eLEE strength is at $x_{\text{min}}=0$. The Feldman-
Cousins 68.3%, 95.5% and 99.7% confidence intervals (CI) are displayed with
the red, green and blue bands, respectively. The 68.3% confidence interval of
the eLEE strength extracted from the MiniBooNE 2020 result are also shown for
comparison purpose.

MicroBooNE 6.369$\times$1020 POT

Figure 29: Distribution of $\Delta\chi^{2}_{\text{nested}}$ assuming eLEEx=1
is true. The sensitivity of rejecting the eLEEx=1 hypothesis is at
2.93$\sigma$ following the Feldman-Cousins procedure. Using the same
procedure, the data measurement rejects eLEEx=1 hypothesis at the p-value of
0.002, which corresponds to 3.14$\sigma$. The results are obtained from 10
million pseudo experiments.

Based on this nested likelihood ratio test statistic, the significance level
of rejecting the eLEEx=1 hypothesis is estimated following the Feldman-Cousins
procedure. As indicated by the black dashed line in Fig. 29, the
$\Delta\chi^{2}_{\text{nested}}$ value of the data measurement is 12.977.
Comparing to the $\Delta\chi^{2}_{\text{nested}}$ distribution from the pseudo
experiments assuming the eLEEx=1 hypothesis is true, the eLEEx=1 hypothesis is
disfavored at a p-value of 0.002, which corresponds to a 3.14$\sigma$
significance level. In comparison with the data result, the sensitivity of
rejecting the eLEEx=1 hypothesis is also calculated using the Asimov data set
of the eLEEx=0 hypothesis (best-fit of eLEE strength, $x_{\text{min}}=0$)
following the procedure in Ref. [106]. The constraint on FC and PC $\nu_{e}$CC
channels from the other five channels are taken into account in the Asimov
data set. The resulting sensitivity, as indicated by the red dashed line in
Fig. 29, is that the eLEEx=1 hypothesis, which represents the median of the
MiniBooNE result, is disfavored at 2.93$\sigma$ with the
$\Delta\chi^{2}_{\text{nested}}$ value of 11.26. Various hypothesis test
results as well as sensitivity values (using the Asimov data set) are
summarized in Table 6. This nested likelihood ratio hypothesis test obtains a
similar level of rejection of the eLEEx=1 hypothesis as the aforementioned
simple-vs-simple likelihood ratio hypothesis test, and provides a rigorous
cross-check of the result.

Scenario | Sensitivity | Data result
---|---|---
$\Delta\chi^{2}_{\text{simple}}$ $\begin{cases}\text{null hypothesis eLEE${}_{x=0}$}\\\ \text{null hypothesis eLEE${}_{x=1}$}\end{cases}$ |  0.25$\sigma$ 3.54$\sigma$ |  0.45$\sigma$ 3.75$\sigma$
$\Delta\chi^{2}_{\text{nested}}$ with null hypothesis eLEEx=1 | 2.93$\sigma$ | 3.14$\sigma$
Table 6: Summary of the hypothesis test results. The simple vs. simple
$\Delta\chi^{2}_{\text{simple}}$ results are from Fig. 27 and the nested
likelihood ratio $\Delta\chi^{2}_{\text{nested}}$ results are from Fig. 29.
The significance level is calculated based on the pseudo experiments
corresponding to each null hypothesis. The sensitivity is calculated using an
Asimov data set assuming the eLEEx=0 hypothesis is true.

##  VIII Summary

In this article, we report a search for a low-energy excess in $\nu_{e}$ CC
interactions using BNB data in the MicroBooNE experiment. A data-driven LEE
model motivated by the previous observation of an electron-like low-energy
excess (eLEE) from the MiniBooNE neutrino experiment is built to quantify the
search result. With the single MicroBooNE detector, measurements of
$\nu_{\mu}$ CC interactions and CC or NC interactions with a $\pi^{0}$ in the
final state are used to constrain the prediction of $\nu_{e}$ CC interactions
as well as other neutrino backgrounds reducing the systematic uncertainty and
thus maximizing the eLEE search sensitivity.

A blind analysis scheme was adopted in analyzing the data with a BNB exposure
of 6.369$\times$1020 POT, which corresponds to the first three years of
MicroBooNE data taking. After unblinding the $\nu_{e}$CC events with
reconstructed neutrino energy below 600 MeV and using a nested likelihood
ratio test statistic using all available information, the best-fit LEE
strength is determined to be 0 (eLEEx=0) with the Feldman-Cousins 68.3%,
95.5%, and 99.7% C.L. upper limits at 0.217, 0.513, and 0.911, respectively.
Using a simple-vs-simple hypothesis test, the eLEEx=1 hypothesis is rejected
at 3.75$\sigma$, while the eLEEx=0 hypothesis is shown to be consistent with
the observation at 0.45$\sigma$. Various hypothesis test results can be found
in Table 6.

Regarding the $\nu_{e}$CC hypothesis to explain the electron-like low-energy
excess observed in the MiniBooNE experiment, we compare the confidence
intervals obtained from the MicroBooNE and MiniBooNE experiments in the
context of the eLEE model. The MiniBooNE 1$\sigma$ statistical uncertainty
band is entirely outside the 3$\sigma$ allowed range of the LEE strength
parameter $x$ derived from this analysis in MicroBooNE. Even in the absence of
any consideration of cross-experiment correlations, the MiniBooNE 1$\sigma$
full uncertainty band is well outside the 2$\sigma$ allowed $x$ parameter
range from MicroBooNE. We should note that the current eLEE model only takes
into account the excess as a function of true neutrino energy with other
kinematics modeled as for the BNB intrinsic $\nu_{e}$ CC events. If we
consider the lepton kinematic distributions reported by MiniBooNE [21] with an
enhanced excess in the forward direction, the achieved $\nu_{e}$ CC selection
efficiency in this work, which is better at forward angles, would yield a
stronger exclusion of the eLEE signal.

While we observe a null result in searching for a $\nu_{e}$ LEE signal using
the current available data set, further tests of alternative hypotheses
explaining the MiniBooNE low-energy excess using single-photon-like events due
to processes either within or beyond the standard model [107, 108, 109, 110,
111] will be carried out and reported in future MicroBooNE publications. While
this analysis did not perform a fit using a sterile-neutrino oscillation
model, future analyses in MicroBooNE with a full data set at 12.25$\times$1020
POT exposure, as well as the short-baseline neutrino program [41], will
explicitly test short-baseline oscillation models in both appearance and
disappearance channels.

###### Acknowledgements.

This document was prepared by the MicroBooNE collaboration using the resources
of the Fermi National Accelerator Laboratory (Fermilab), a U.S. Department of
Energy, Office of Science, HEP User Facility. Fermilab is managed by Fermi
Research Alliance, LLC (FRA), acting under Contract No. DE-AC02-07CH11359.
MicroBooNE is supported by the following: the U.S. Department of Energy,
Office of Science, Offices of High Energy Physics and Nuclear Physics; the
U.S. National Science Foundation; the Swiss National Science Foundation; the
Science and Technology Facilities Council (STFC), part of the United Kingdom
Research and Innovation; the Royal Society (United Kingdom); and The European
Union’s Horizon 2020 Marie Sklodowska-Curie Actions. Additional support for
the laser calibration system and cosmic ray tagger was provided by the Albert
Einstein Center for Fundamental Physics, Bern, Switzerland. We also
acknowledge the contributions of technical and scientific staff to the design,
construction, and operation of the MicroBooNE detector as well as the
contributions of past collaborators to the development of MicroBooNE analyses,
without whom this work would not have been possible.

*

##  APPENDIX A Separation of 0$p$X$\pi$ and N$p$X$\pi$ samples

MicroBooNEFC, 0$p$X$\pi$

(a) FC $\nu_{\mu}$ CC, 0$p$X$\pi$

MicroBooNEPC, 0$p$X$\pi$

(b) PC $\nu_{\mu}$ CC, 0$p$X$\pi$

MicroBooNEFC, N$p$X$\pi$

(c) FC $\nu_{\mu}$ CC, N$p$X$\pi$

MicroBooNEPC, N$p$X$\pi$

(d) PC $\nu_{\mu}$ CC, N$p$X$\pi$

Figure 30: Event distributions of FC $\nu_{\mu}$ CC 0$p$X$\pi$, PC $\nu_{\mu}$
CC 0$p$X$\pi$, FC $\nu_{\mu}$ CC N$p$X$\pi$, and PC $\nu_{\mu}$ CC N$p$X$\pi$
samples in figures (a)-(d), respectively. The breakdown of each component for
different final states for both signal and background events is shown in the
legend (see definitions in Sec. IV.1). The bottom sub-panels present the data-
prediction ratios as well as the statistical and systematic uncertainties. The
pink band includes the MC statistical, cross section, and flux uncertainties.
The purple band corresponds to the full uncertainty with an addition of
detector systematic uncertainty. No constraint is applied.

MicroBooNEFC, 0$p$X$\pi$

(a) FC $\nu_{e}$ CC, 0$p$X$\pi$

MicroBooNEPC, 0$p$X$\pi$

(b) PC $\nu_{e}$ CC, 0$p$X$\pi$

MicroBooNEFC, N$p$X$\pi$

(c) FC $\nu_{e}$ CC, N$p$X$\pi$

MicroBooNEPC, N$p$X$\pi$

(d) PC $\nu_{e}$ CC, N$p$X$\pi$

Figure 31: Event distributions of FC $\nu_{e}$ CC 0$p$X$\pi$, PC $\nu_{e}$ CC
0$p$X$\pi$, FC $\nu_{e}$ CC N$p$X$\pi$, and PC $\nu_{e}$ CC N$p$X$\pi$ samples
in figures (a)-(d), respectively. The breakdown of each component for
different final states for both signal and background events is shown in the
legend (see definitions in Sec. IV.1). The bottom sub-panels present the data-
prediction ratios as well as the statistical and systematic uncertainties. The
pink band includes the MC statistical, cross section, and flux uncertainties.
The purple band corresponds to the full uncertainty with an addition of
detector systematic uncertainty. No constraint is applied.

As shown in Fig. 22 and discussed in Sec. VI.3, the enhancement of the
$\nu_{\mu}$ CC events with low hadronic energies can be adequately described
by the simulation, in particular by the cross-section model and its allowed
variation which is most relevant to the potential missing energy in the
hadronic system, employed in the MicroBooNE MC simulation.

To further validate the reconstruction of interactions with inclusive hadronic
final states and cross-check its impact on the analysis results, the inclusive
$\nu_{\mu}$ CC events (Fig. 21(a) and 21(b)) are further divided into
0$p$X$\pi$ and N$p$X$\pi$ (N$\geq$1) channels where $p$ is defined as
reconstructed protons with kinetic energy greater than 35 MeV (corresponding
to a length of 1 cm) and X is the number of reconstructed pions (X$\geq$0).
The separated 0$p$X$\pi$ and N$p$X$\pi$ channels for the selected $\nu_{\mu}$
CC events are shown in Fig. 30. Interestingly, the enhancement of $\nu_{\mu}$
CC events is found to show up in the 0$p$X$\pi$ channel, whereas the
N$p$X$\pi$ channel has an overall good agreement between the data and MC
prediction. This suggests that the enhancement of the inclusive $\nu_{\mu}$ CC
events in the relatively low energy region is less likely to be caused by
potential bias in the neutrino flux prediction. Based on the nominal MC
simulation, the $\nu_{\mu}$ CC 0$p$X$\pi$ FC+PC events, excluding true
$\nu_{e}$ CC and NC events, have 41% (46%) true 0$p$X$\pi$ events, and the
N$p$X$\pi$ FC+PC events have 96% (97%) true N$p$X$\pi$ events in the full
energy region (low energy region of $E^{\rm rec}_{\nu}<$ 600 MeV). The kinetic
energy threshold of the protons in the truth-level study is 40 MeV. To
complement the above truth-level studies which indicate contamination of
N$p$X$\pi$ events in the 0$p$X$\pi$ $\nu_{\mu}$ CC selections, we assess from
hand-scans of both data and MC simulation that protons which contaminate the
selection are generally low energy. It supports the statement that the
$\nu_{\mu}$ CC channels successfully split interactions by the amount of
proton hadronic activity, consistent with the goal of this study. This
indicates the excess 0$p$X$\pi$ events are not due to the migration of
N$p$X$\pi$ events with high energy protons. Finally, the hypothesis of
migration of high energy events into the low energy region is disfavored
because the deficit in the high energy region for both 0$p$X$\pi$ and
N$p$X$\pi$ $\nu_{\mu}$ CC channels has insufficient events to account for the
excess in the low energy region.

The inclusive $\nu_{e}$ CC events (Fig. 24) are divided into 0$p$X$\pi$ and
N$p$X$\pi$ channels as well in order to investigate the consistency between
$\nu_{\mu}$ CC and $\nu_{e}$ CC selections assuming the enhancement of
0$p$X$\pi$ events is largely attributed to the cross section modeling. The
separated 0$p$X$\pi$ and N$p$X$\pi$ channels for the selected $\nu_{e}$ CC
events are shown in Fig. 31, with no constraints applied in the MC
predictions. Based on a truth-level study similar to that for $\nu_{\mu}$ CC
events, the $\nu_{e}$ CC 0$p$X$\pi$ FC+PC events, excluding true $\nu_{\mu}$
CC and NC events, have 39% (49%) true 0$p$X$\pi$ events, and the N$p$X$\pi$
FC+PC events have 94% (98%) true N$p$X$\pi$ events in the full energy region
(low energy region of $E^{\rm rec}_{\nu}<$ 600 MeV). The data-prediction ratio
in the relatively low energy region of the $\nu_{e}$ CC 0$p$X$\pi$ channel
appears higher than that in the $\nu_{e}$ CC N$p$X$\pi$ channel, in analogy to
the $\nu_{\mu}$ CC results.

MicroBooNE 6.369$\times$1020 POT

Figure 32:  $\Delta\chi^{2}_{\rm nested}$ value as a function of eLEE
strength. “7-ch” (black curve) corresponds to the result from the default
7-channel simultaneous fit which is the same as the one in Fig. 28. “11-ch”
(green curve) corresponds to the result from a 11-channel simultaneous fit
combining separated 0$p$X$\pi$ and N$p$X$\pi$ channels for $\nu_{e}$ CC and
$\nu_{\mu}$ CC FC and PC events together with the dedicated $\pi^{0}$
channels. “0pX$\pi$” (red curve) corresponds to the result from the 0$p$X$\pi$
channels for $\nu_{e}$ CC and $\nu_{\mu}$ CC FC and PC events as well as the
dedicated $\pi^{0}$ channels. “NpX$\pi$” (blue curve) corresponds to the
result from the N$p$X$\pi$ channels for $\nu_{e}$ CC and $\nu_{\mu}$ CC FC and
PC events as well as the dedicated $\pi^{0}$ channels.

In order to understand the impact of, and account for the observed behavior in
the data-prediction ratios in the 0$p$X$\pi$ and N$p$X$\pi$ channels on the
eLEE analysis, the eLEE strength fit is repeated separating the inclusive
$\nu_{\mu}$ CC and $\nu_{e}$ CC channels by these exclusive final-state
topologies. The default 7 channels used in the eLEE analysis are expanded to
11 channels where each $\nu_{\mu}$ CC or $\nu_{e}$ CC channel is separated
into two (0$p$X$\pi$ and N$p$X$\pi$). The results can be found in Fig. 32. The
11-channel result combining separated 0$p$X$\pi$ and N$p$X$\pi$ channels is
well consistent with the default 7-channel result. Furthermore, the individual
0$p$X$\pi$ or N$p$X$\pi$ results also have best-fit eLEE strengths at zero
albeit the uncertainties and sensitivities are different. Therefore, the
consistency between $\nu_{\mu}$ CC and $\nu_{e}$ CC selections with respect to
the different hadronic final states is validated, and it supports the
inclusive 7-channel fit strategy as well as the overall conclusion of this
paper. Additional work studying $\nu_{\mu}$ CC interaction cross-sections in
exclusive final states is ongoing within the collaboration with the goal of
further improving our ability to model these processes.

## References

  * Zyla  _et al._ [2020] P. A. Zyla _et al._ (Particle Data Group), Review of Particle Physics, PTEP 2020, 083C01 (2020).
  * Aker  _et al._ [2019] M. Aker _et al._ (KATRIN Collaboration), Improved Upper Limit on the Neutrino Mass from a Direct Kinematic Method by KATRIN, Phys. Rev. Lett. 123, 221802 (2019), arXiv:1909.06048 [hep-ex] .
  * King [2004] S. F. King, Neutrino mass models, Rept. Prog. Phys. 67, 107 (2004), arXiv:hep-ph/0310204 .
  * Schael  _et al._ [2006] S. Schael _et al._ (ALEPH, DELPHI, L3, OPAL, SLD, LEP Electroweak Working Group, SLD Electroweak Group, SLD Heavy Flavour Group), Precision electroweak measurements on the $Z$ resonance, Phys. Rept. 427, 257 (2006), arXiv:hep-ex/0509008 .
  * Pontecorvo [1967] B. Pontecorvo, Neutrino Experiments and the Problem of Conservation of Leptonic Charge, Zh. Eksp. Teor. Fiz. 53, 1717 (1967).
  * Adamson  _et al._ [2020] P. Adamson _et al._ (MINOS+, Daya Bay Collaboration), Improved Constraints on Sterile Neutrino Mixing from Disappearance Searches in the MINOS, MINOS+, Daya Bay, and Bugey-3 Experiments, Phys. Rev. Lett. 125, 071801 (2020), arXiv:2002.00301 [hep-ex] .
  * Andriamirado  _et al._ [2021] M. Andriamirado _et al._ (PROSPECT Collaboration), Improved short-baseline neutrino oscillation search and energy spectrum measurement with the PROSPECT experiment at HFIR, Phys. Rev. D 103, 032001 (2021), arXiv:2006.11210 [hep-ex] .
  * Kaether  _et al._ [2010] F. Kaether, W. Hampel, G. Heusser, J. Kiko, and T. Kirsten, Reanalysis of the GALLEX solar neutrino flux and source experiments, Phys. Lett. B 685, 47 (2010), arXiv:1001.2731 [hep-ex] .
  * Barinov  _et al._ [2021] V. V. Barinov _et al._ , Results from the Baksan Experiment on Sterile Transitions (BEST) (2021) arXiv:2109.11482 [nucl-ex] .
  * Abdurashitov  _et al._ [2009] J. N. Abdurashitov _et al._ (SAGE Collaboration), Measurement of the solar neutrino capture rate with gallium metal. III: Results for the 2002–2007 data-taking period, Phys. Rev. C 80, 015807 (2009), arXiv:0901.2200 [nucl-ex] .
  * Mention  _et al._ [2011] G. Mention, M. Fechner, T. Lasserre, T. A. Mueller, D. Lhuillier, M. Cribier, and A. Letourneau, The Reactor Antineutrino Anomaly, Phys. Rev. D 83, 073006 (2011), arXiv:1101.2755 [hep-ex] .
  * Huber [2011] P. Huber, On the determination of anti-neutrino spectra from nuclear reactors, Phys. Rev. C 84, 024617 (2011), [Erratum: Phys.Rev.C 85, 029901 (2012)], arXiv:1106.0687 [hep-ph] .
  * Mueller  _et al._ [2011] T. A. Mueller _et al._ , Improved Predictions of Reactor Antineutrino Spectra, Phys. Rev. C 83, 054615 (2011), arXiv:1101.2663 [hep-ex] .
  * An  _et al._ [2017] F. P. An _et al._ (Daya Bay), Evolution of the Reactor Antineutrino Flux and Spectrum at Daya Bay, Phys. Rev. Lett. 118, 251801 (2017), arXiv:1704.01082 [hep-ex] .
  * An  _et al._ [2021] F. P. An _et al._ (Daya Bay), Antineutrino energy spectrum unfolding based on the Daya Bay measurement and its applications, Chin. Phys. C 45, 073001 (2021), arXiv:2102.04614 [hep-ex] .
  * Estienne  _et al._ [2019] M. Estienne _et al._ , Updated Summation Model: An Improved Agreement with the Daya Bay Antineutrino Fluxes, Phys. Rev. Lett. 123, 022502 (2019), arXiv:1904.09358 [nucl-ex] .
  * Giunti  _et al._ [2022] C. Giunti, Y. F. Li, C. A. Ternes, and Z. Xin, Reactor antineutrino anomaly in light of recent flux model refinements, Phys. Lett. B 829, 137054 (2022), arXiv:2110.06820 [hep-ph] .
  * Serebrov  _et al._ [2021] A. P. Serebrov _et al._ , Search for sterile neutrinos with the Neutrino-4 experiment and measurement results, Phys. Rev. D 104, 032003 (2021), arXiv:2005.05301 [hep-ex] .
  * Aguilar-Arevalo  _et al._ [2001] A. Aguilar-Arevalo _et al._ (LSND Collaboration), Evidence for neutrino oscillations from the observation of $\bar{\nu}_{e}$ appearance in a $\bar{\nu}_{\mu}$ beam, Phys. Rev. D 64, 112007 (2001), arXiv:hep-ex/0104049 .
  * Aguilar-Arevalo  _et al._ [2013] A. A. Aguilar-Arevalo _et al._ (MiniBooNE Collaboration), Improved Search for $\bar{\nu}_{\mu}\rightarrow\bar{\nu}_{e}$ Oscillations in the MiniBooNE Experiment, Phys. Rev. Lett. 110, 161801 (2013), arXiv:1303.2588 [hep-ex] .
  * Aguilar-Arevalo  _et al._ [2021] A. A. Aguilar-Arevalo _et al._ (MiniBooNE Collaboration), Updated MiniBooNE neutrino oscillation results with increased data and new background studies, Phys. Rev. D 103, 052002 (2021), arXiv:2006.16883 [hep-ex] .
  * Giunti and Lasserre [2019] C. Giunti and T. Lasserre, eV-scale Sterile Neutrinos, Ann. Rev. Nucl. Part. Sci. 69, 163 (2019), arXiv:1901.08330 [hep-ph] .
  * Willis and Radeka [1974] W. Willis and V. Radeka, Liquid Argon Ionization Chambers as Total Absorption Detectors, Nucl. Instrum. Meth. 120, 221 (1974).
  * Rubbia [1977] C. Rubbia, The Liquid Argon Time Projection Chamber: A New Concept for Neutrino Detectors, CERN-EP-INT-77-08, CERN-EP-77-08 (1977).
  * Chen  _et al._ [1976] H. Chen, P. Condon, B. Barish, and F. Sciulli, A Neutrino detector sensitive to rare processes. I. A Study of neutrino electron reactions, FERMILAB-PROPOSAL-0496 (1976).
  * Nygren [1974] D. Nygren, The Time Projection Chamber: A New 4 pi Detector for Charged Particles, eConf C740805, 58 (1974).
  * Amerio  _et al._ [2004] S. Amerio _et al._ (ICARUS), Design, construction and tests of the ICARUS T600 detector, Nucl. Instrum. Meth. A 527, 329 (2004).
  * Anderson  _et al._ [2012] C. Anderson _et al._ , The ArgoNeuT Detector in the NuMI Low-Energy beam line at Fermilab, JINST 7, P10019, arXiv:1205.6747 [physics.ins-det] .
  * Berns  _et al._ [2013] H. Berns _et al._ (CAPTAIN), The CAPTAIN Detector and Physics Program, in  _Community Summer Study 2013: Snowmass on the Mississippi_ (2013) arXiv:1309.1740 [physics.ins-det] .
  * Hahn  _et al._ [2016] A. Hahn _et al._ (LBNE Collaboration), The LBNE 35 Ton Prototype Cryostat, in  _2014 IEEE Nuclear Science Symposium and Medical Imaging Conference and 21st Symposium on Room-Temperature Semiconductor X-ray and Gamma-ray Detectors_ (2016) p. 7431158.
  * Cavanna  _et al._ [2014] F. Cavanna, M. Kordosky, J. Raaf, and B. Rebel (LArIAT), LArIAT: Liquid Argon In A Testbeam (2014) arXiv:1406.5560 [physics.ins-det] .
  * Acciarri  _et al._ [2017a] R. Acciarri _et al._ (MicroBooNE Collaboration), Design and Construction of the MicroBooNE Detector, JINST 12 (02), P02017.
  * Abud  _et al._ [2022] A. A. Abud _et al._ (DUNE), Design, construction and operation of the ProtoDUNE-SP Liquid Argon TPC, JINST 17 (01), P01005, arXiv:2108.01902 [physics.ins-det] .
  * Cavanna  _et al._ [2018] F. Cavanna, A. Ereditato, and B. T. Fleming, Advances in liquid argon detectors, Nucl. Instrum. Meth. A907, 1 (2018).
  * Aguilar-Arevalo  _et al._ [2009a] A. A. Aguilar-Arevalo _et al._ (MiniBooNE Collaboration), The Neutrino Flux prediction at MiniBooNE, Phys. Rev. D 79, 072002 (2009a), arXiv:0806.1449 [hep-ex] .
  * Abratenko  _et al._ [2021a] P. Abratenko _et al._ (MicroBooNE Collaboration), Search for a Higgs Portal Scalar Decaying to Electron-Positron Pairs in the MicroBooNE Detector, Phys. Rev. Lett. 127, 151803 (2021a), arXiv:2106.00568 [hep-ex] .
  * Abratenko  _et al._ [2022a] P. Abratenko _et al._ (MicroBooNE), Search for Neutrino-Induced Neutral-Current $\Delta$ Radiative Decay in MicroBooNE and a First Test of the MiniBooNE Low Energy Excess under a Single-Photon Hypothesis, Phys. Rev. Lett. 128, 111801 (2022a), arXiv:2110.00409 [hep-ex] .
  * Abratenko  _et al._ [2021b] P. Abratenko _et al._ (MicroBooNE Collaboration), Search for an anomalous excess of charged-current quasi-elastic $\nu_{e}$ interactions with the MicroBooNE experiment using Deep-Learning-based reconstruction (2021) (accepted by PRD), arXiv:2110.14080 [hep-ex] .
  * Abratenko  _et al._ [2021c] P. Abratenko _et al._ (MicroBooNE Collaboration), Search for an anomalous excess of charged-current $\nu_{e}$ interactions without pions in the final state with the MicroBooNE experiment (2021) (accepted by PRD), arXiv:2110.14065 [hep-ex] .
  * Abratenko  _et al._ [2021d] P. Abratenko _et al._ (MicroBooNE Collaboration), Search for an Excess of Electron Neutrino Interactions in MicroBooNE Using Multiple Final State Topologies (2021) (accepted by PRL), arXiv:2110.14054 [hep-ex] .
  * Antonello  _et al._ [2015] M. Antonello _et al._ (MicroBooNE, LAr1-ND, and ICARUS-WA104 Collaboration), A Proposal for a Three Detector Short-Baseline Neutrino Oscillation Program in the Fermilab Booster Neutrino Beam, arXiv:1503.01520 [physics.ins-det] (2015).
  * Qian  _et al._ [2018] X. Qian, C. Zhang, B. Viren, and M. Diwan, Three-dimensional Imaging for Large LArTPCs, JINST 13 (05), P05032, arXiv:1803.04850 [physics.ins-det] .
  * Abratenko  _et al._ [2021e] P. Abratenko _et al._ (MicroBooNE Collaboration), Neutrino event selection in the MicroBooNE liquid argon time projection chamber using Wire-Cell 3D imaging, clustering, and charge-light matching, JINST 16 (06), P06043, arXiv:2011.01375 [physics.ins-det] .
  * Abratenko  _et al._ [2018] P. Abratenko _et al._ (MicroBooNE Collaboration), Booster Neutrino Flux Prediction at MicroBooNE (2018), MICROBOONE-NOTE-1031-PUB.
  * Andreopoulos  _et al._ [2010] C. Andreopoulos _et al._ , The GENIE Neutrino Monte Carlo Generator, Nucl. Instrum. Meth. A 614, 87 (2010), arXiv:0905.2517 [hep-ph] .
  * Alvarez-Ruso  _et al._ [2021] L. Alvarez-Ruso _et al._ (GENIE), Recent highlights from GENIE v3 (2021) arXiv:2106.09381 [hep-ph] .
  * Adams  _et al._ [2018a] C. Adams _et al._ (MicroBooNE Collaboration), Ionization electron signal processing in single phase LArTPCs. Part I. Algorithm Description and quantitative evaluation with MicroBooNE simulation, JINST 13 (07), P07006.
  * Abratenko  _et al._ [2020a] P. Abratenko _et al._ (MicroBooNE Collaboration), Measurement of Space Charge Effects in the MicroBooNE LArTPC Using Cosmic Muons, JINST 15, P12037.
  * Adams  _et al._ [2020a] C. Adams _et al._ (MicroBooNE Collaboration), Calibration of the Charge and Energy Response of the MicroBooNE Liquid Argon Time Projection Chamber using Muons and Protons, JINST 15 (03), P03022.
  * Conrad and Shaevitz [2018] J. M. Conrad and M. H. Shaevitz, Sterile Neutrinos: An Introduction to Experiments, Adv. Ser. Direct. High Energy Phys. 28, 391 (2018), arXiv:1609.07803 [hep-ex] .
  * Conrad  _et al._ [2013] J. M. Conrad, W. C. Louis, and M. H. Shaevitz, The LSND and MiniBooNE Oscillation Searches at High $\Delta m^{2}$, Ann. Rev. Nucl. Part. Sci. 63, 45 (2013), arXiv:1306.6494 [hep-ex] .
  * Aguilar-Arevalo  _et al._ [2018] A. A. Aguilar-Arevalo _et al._ (MiniBooNE Collaboration), Significant Excess of Electron-Like Events in the MiniBooNE Short-Baseline Neutrino Experiment, Phys. Rev. Lett. 121, 221801 (2018), arXiv:1805.12028 [hep-ex] .
  * D’Agostini [1995] G. D’Agostini, A Multidimensional unfolding method based on Bayes’ theorem, Nucl. Instrum. Meth. A 362, 487 (1995).
  * Ji  _et al._ [2020] X. Ji, W. Gu, X. Qian, H. Wei, and C. Zhang, Combined Neyman–Pearson chi-square: An improved approximation to the Poisson-likelihood chi-square, Nucl. Instrum. Meth. A 961, 163677 (2020), arXiv:1903.07185 [physics.data-an] .
  * Feldman and Cousins [1998] G. J. Feldman and R. D. Cousins, A Unified approach to the classical statistical analysis of small signals, Phys. Rev. D 57, 3873 (1998), arXiv:physics/9711021 .
  * Eaton [1983] M. L. Eaton, Multivariate statistics: a vector space approach, John Wiley and Sons , 116 (1983).
  * Abratenko  _et al._ [2021f] P. Abratenko _et al._ (MicroBooNE Collaboration), Cosmic Ray Background Rejection with Wire-Cell LArTPC Event Reconstruction in the MicroBooNE Detector, Phys. Rev. Applied 15, 064071 (2021f), arXiv:2101.05076 [physics.ins-det] .
  * Adams  _et al._ [2020b] C. Adams _et al._ (MicroBooNE Collaboration), A Method to Determine the Electric Field of Liquid Argon Time Projection Chambers Using a UV Laser System and its Application in MicroBooNE, JINST 15 (07), P07010.
  * Agostinelli  _et al._ [2003] S. Agostinelli _et al._ (GEANT4), GEANT4–a simulation toolkit, Nucl. Instrum. Meth. A 506, 250 (2003).
  * Del Tutto [2017] M. Del Tutto, VENu: The Virtual Environment for Neutrinos, in  _Meeting of the APS Division of Particles and Fields_ (2017) arXiv:1709.10120 [physics.pop-ph] .
  * Briese  _et al._ [2013] T. Briese _et al._ , Testing of Cryogenic Photomultiplier Tubes for the MicroBooNE Experiment, JINST 8, T07005.
  * Li  _et al._ [2016] Y. Li _et al._ (MicroBooNE Collaboration), Measurement of Longitudinal Electron Diffusion in Liquid Argon, Nucl. Instrum. Meth. A816, 160 (2016).
  * Radeka  _et al._ [2011] V. Radeka _et al._ , Cold electronics for ’Giant’ Liquid Argon Time Projection Chambers,  _Giant liquid argon charge imaging experiment. Proceedings, 1st International Workshop, GLA2010, Tsukuba, Japan, March 29-31, 2010_ , J. Phys. Conf. Ser. 308, 012021 (2011).
  * Acciarri  _et al._ [2017b] R. Acciarri _et al._ (MicroBooNE Collaboration), Noise Characterization and Filtering in the MicroBooNE Liquid Argon TPC, JINST 12 (08), P08003.
  * Nieves  _et al._ [2012] J. Nieves, I. Ruiz Simo, and M. J. Vicente Vacas, The nucleon axial mass and the MiniBooNE Quasielastic Neutrino-Nucleus Scattering problem, Phys. Lett. B 707, 72 (2012), arXiv:1106.5374 [hep-ph] .
  * Nieves  _et al._ [2004] J. Nieves, J. E. Amaro, and M. Valverde, Inclusive quasi-elastic neutrino reactions, Phys. Rev. C 70, 055503 (2004), [Erratum: Phys.Rev.C 72, 019902 (2005)], arXiv:nucl-th/0408005 .
  * Gran  _et al._ [2013] R. Gran, J. Nieves, F. Sanchez, and M. J. Vicente Vacas, Neutrino-nucleus quasi-elastic and 2p2h interactions up to 10 GeV, Phys. Rev. D 88, 113007 (2013), arXiv:1307.8105 [hep-ph] .
  * Abratenko  _et al._ [2019] P. Abratenko _et al._ (MicroBooNE Collaboration), First Measurement of Inclusive Muon Neutrino Charged Current Differential Cross Sections on Argon at $E_{\nu}\sim$0.8 GeV with the MicroBooNE Detector, Phys. Rev. Lett. 123, 131801 (2019), arXiv:1905.09694 [hep-ex] .
  * Abratenko  _et al._ [2020b] P. Abratenko _et al._ (MicroBooNE Collaboration), First Measurement of Differential Charged Current Quasielastic-like $\nu_{\mu}$-Argon Scattering Cross Sections with the MicroBooNE Detector, Phys. Rev. Lett. 125, 201803 (2020b), arXiv:2006.00108 [hep-ex] .
  * Abratenko  _et al._ [2020c] P. Abratenko _et al._ (MicroBooNE Collaboration), Measurement of differential cross sections for $\nu_{\mu}$ -Ar charged-current interactions with protons and no pions in the final state with the MicroBooNE detector, Phys. Rev. D 102, 112013 (2020c), arXiv:2010.02390 [hep-ex] .
  * Abratenko  _et al._ [2021g] P. Abratenko _et al._ (MicroBooNE Collaboration), Measurement of the flux-averaged inclusive charged-current electron neutrino and antineutrino cross section on argon using the NuMI beam and the MicroBooNE detector, Phys. Rev. D 104, 052002 (2021g), arXiv:2101.04228 [hep-ex] .
  * Abe  _et al._ [2016] K. Abe _et al._ (T2K Collaboration), Measurement of double-differential muon neutrino charged-current interactions on C8H8 without pions in the final state using the T2K off-axis beam, Phys. Rev. D 93, 112012 (2016), arXiv:1602.03652 [hep-ex] .
  * Abratenko  _et al._ [2022b] P. Abratenko _et al._ (MicroBooNE), New $CC0\pi$ GENIE model tune for MicroBooNE, Phys. Rev. D 105, 072001 (2022b), arXiv:2110.14028 [hep-ex] .
  * Snider and Petrillo [2017] E. L. Snider and G. Petrillo, LArSoft: Toolkit for Simulation, Reconstruction and Analysis of Liquid Argon TPC Neutrino Detectors, J. Phys. Conf. Ser. 898, 042057 (2017).
  * Jaskolski and Wojcik [2011] M. Jaskolski and M. Wojcik, Electron Recombination in Ionized Liquid Argon: A Computational Approach Based on Realistic Models of Electron Transport and Reactions, J. Phys. Chem. A 115, 4317 (2011).
  * Acciarri  _et al._ [2013] R. Acciarri _et al._ (ArgoNeuT Collaboration), A Study of Electron Recombination Using Highly Ionizing Particles in the ArgoNeuT Liquid Argon TPC, JINST 8, P08005, arXiv:1306.1712 [physics.ins-det] .
  * Veenhof [1998] R. Veenhof, GARFIELD, recent developments, Nucl. Instrum. Meth. A 419, 726 (1998).
  * Ramo [1939] S. Ramo, Currents induced by electron motion, Proc. Ire. 27, 584 (1939).
  * Adams  _et al._ [2018b] C. Adams _et al._ (MicroBooNE Collaboration), Ionization electron signal processing in single phase LArTPCs. Part II. Data/simulation comparison and performance in MicroBooNE, JINST 13 (07), P07007.
  * Acciarri  _et al._ [2017c] R. Acciarri _et al._ (MicroBooNE Collaboration), Measurement of cosmic-ray reconstruction efficiencies in the MicroBooNE LArTPC using a small external cosmic-ray counter, JINST 12 (12), P12030.
  * Adams  _et al._ [2019a] C. Adams _et al._ (MicroBooNE Collaboration), Comparison of $\nu_{\mu}$-Ar multiplicity distributions observed by MicroBooNE to GENIE model predictions, Eur. Phys. J. C79, 248 (2019a).
  * Adams  _et al._ [2019b] C. Adams _et al._ (MicroBooNE Collaboration), First measurement of $\nu_{\mu}$ charged-current $\pi^{0}$ production on argon with the MicroBooNE detector, Phys. Rev. D99, 091102 (2019b).
  * Adams  _et al._ [2019c] C. Adams _et al._ (MicroBooNE Collaboration), Rejecting cosmic background for exclusive charged current quasi elastic neutrino interaction studies with Liquid Argon TPCs; a case study with the MicroBooNE detector, Eur. Phys. J. C79, 673 (2019c).
  * Abratenko  _et al._ [2020d] P. Abratenko _et al._ (MicroBooNE Collaboration), High-performance Generic Neutrino Detection in a LArTPC near the Earth’s Surface with the MicroBooNE Detector (2020) arXiv:2012.07928 [hep-ex] .
  * Adams  _et al._ [2019d] C. Adams _et al._ (MicroBooNE Collaboration), Design and construction of the MicroBooNE Cosmic Ray Tagger system, JINST 14 (04), P04004.
  * Abratenko  _et al._ [2022c] P. Abratenko _et al._ (MicroBooNE Collaboration), Wire-Cell 3D Pattern Recognition Techniques for Neutrino Event Reconstruction in Large LArTPCs: Algorithm Description and Quantitative Evaluation with MicroBooNE Simulation, JINST 17, P01037, arXiv:2110.13961 [hep-ex] .
  * Graham  _et al._ [2018] B. Graham, M. Engelcke, and L. van der Maaten, 3d semantic segmentation with submanifold sparse convolutional networks, CVPR 10.1109/CVPR.2018.00961 (2018).
  * pst [2019] PSTAR at NIST, https://physics.nist.gov/PhysRefData/Star/Text/PSTAR.html (2019).
  * Shibamura  _et al._ [1975] E. Shibamura, A. Hitachi, T. Doke, T. Takahashi, S. Kubota, and M. Miyajima, Drift velocities of electrons, saturation characteristics of ionization and W-values for conversion electrons in liquid argon, liquid argon-gas mixtures and liquid xenon, Nucl. Instrum. Meth. 131, 249 (1975).
  * Miyajima  _et al._ [1974] M. Miyajima, T. Takahashi, S. Konno, T. Hamada, S. Kubota, H. Shibamura, and T. Doke, Average energy expended per ion pair in liquid argon, Phys. Rev. A 9, 1438 (1974).
  * Chen and Guestrin [2016] T. Chen and C. Guestrin, XGBoost: A Scalable Tree Boosting System (2016) arXiv:1603.02754 [cs.LG] .
  * Adamson  _et al._ [2016] P. Adamson _et al._ , The NuMI Neutrino Beam, Nucl. Instrum. Meth. A806, 279 (2016).
  * Cheng  _et al._ [2011] G. Cheng _et al._ (SciBooNE Collaboration), Measurement of $K^{+}$ production cross section by 8 GeV protons using high energy neutrino interactions in the SciBooNE detector, Phys. Rev. D 84, 012009 (2011), arXiv:1105.2871 [hep-ex] .
  * Mariani  _et al._ [2011] C. Mariani, G. Cheng, J. M. Conrad, and M. H. Shaevitz, Improved Parameterization of $K^{+}$ Production in p-Be Collisions at Low Energy Using Feynman Scaling, Phys. Rev. D 84, 114021 (2011), arXiv:1110.0417 [hep-ex] .
  * Catanesi  _et al._ [2007] M. G. Catanesi _et al._ (HARP Collaboration), Measurement of the production cross-section of positive pions in the collision of 8.9-GeV/c protons on beryllium, Eur. Phys. J. C 52, 29 (2007), arXiv:hep-ex/0702024 .
  * Weinberg [1958] S. Weinberg, Charge symmetry of weak interactions, Phys. Rev. 112, 1375 (1958).
  * Day and McFarland [2012] M. Day and K. S. McFarland, Differences in Quasi-Elastic Cross-Sections of Muon and Electron Neutrinos, Phys. Rev. D 86, 053003 (2012), arXiv:1206.6745 [hep-ph] .
  * Wright and Kelsey [2015] D. H. Wright and M. H. Kelsey, The Geant4 Bertini Cascade, Nucl. Instrum. Meth. A 804, 175 (2015).
  * Calcutt  _et al._ [2021] J. Calcutt, C. Thorpe, K. Mahn, and L. Fields, Geant4Reweight: a framework for evaluating and propagating hadronic interaction uncertainties in Geant4, JINST 16 (08), P08042, arXiv:2105.01744 [physics.data-an] .
  * Abratenko  _et al._ [2021h] P. Abratenko _et al._ (MicroBooNE), Novel Approach for Evaluating Detector-Related Uncertainties in a LArTPC Using MicroBooNE Data (2021) (accepted by EJPC), arXiv:2111.03556 [hep-ex] .
  * Chernick  _et al._ [2011] M. R. Chernick, W. Gonzalez-Manteiga, R. M. Crukeiras, and E. B. Barrios, Bootstrap methods, International Encyclopedia of Statistical Science 110.1007/978-3-642-04898-2_150 (2011).
  * Glüsenkamp [2020] T. Glüsenkamp, Unified perspective on modified Poisson likelihoods for limited Monte Carlo data, JINST 15 (01), P01035, arXiv:1902.08831 [astro-ph.IM] .
  * Argüelles  _et al._ [2019] C. A. Argüelles, A. Schneider, and T. Yuan, A binned likelihood for stochastic models, JHEP 06, 030, arXiv:1901.04645 [physics.data-an] .
  * Hauschild and Jentschel [2001] T. Hauschild and M. Jentschel, Comparison of maximum likelihood estimation and chi-square statistics applied to counting experiments, Nucl. Instrum. Meth. A457, 384 (2001).
  * Abratenko  _et al._ [2022d] P. Abratenko _et al._ ((MicroBooNE Collaboration)*, MicroBooNE), First Measurement of Energy-Dependent Inclusive Muon Neutrino Charged-Current Cross Sections on Argon with the MicroBooNE Detector, Phys. Rev. Lett. 128, 151801 (2022d), arXiv:2110.14023 [hep-ex] .
  * Cowan  _et al._ [2011] G. Cowan, K. Cranmer, E. Gross, and O. Vitells, Asymptotic formulae for likelihood-based tests of new physics, Eur. Phys. J. C 71, 1554 (2011), [Erratum: Eur.Phys.J.C 73, 2501 (2013)], arXiv:1007.1727 [physics.data-an] .
  * Aguilar-Arevalo  _et al._ [2009b] A. A. Aguilar-Arevalo _et al._ (MiniBooNE Collaboration), Unexplained Excess of Electron-Like Events From a 1-GeV Neutrino Beam, Phys. Rev. Lett. 102, 101802 (2009b), arXiv:0812.2243 [hep-ex] .
  * Wang  _et al._ [2014] E. Wang, L. Alvarez-Ruso, and J. Nieves, Photon emission in neutral current interactions at intermediate energies, Phys. Rev. C 89, 015503 (2014), arXiv:1311.2151 [nucl-th] .
  * Bertuzzo  _et al._ [2018] E. Bertuzzo, S. Jana, P. A. N. Machado, and R. Zukanovich Funchal, Dark Neutrino Portal to Explain MiniBooNE excess, Phys. Rev. Lett. 121, 241801 (2018), arXiv:1807.09877 [hep-ph] .
  * Ballett  _et al._ [2019] P. Ballett, S. Pascoli, and M. Ross-Lonergan, U(1)’ mediated decays of heavy sterile neutrinos in MiniBooNE, Phys. Rev. D 99, 071701 (2019), arXiv:1808.02915 [hep-ph] .
  * Abdallah  _et al._ [2021] W. Abdallah, R. Gandhi, and S. Roy, Two-Higgs doublet solution to the LSND, MiniBooNE and muon g-2 anomalies, Phys. Rev. D 104, 055028 (2021), arXiv:2010.06159 [hep-ph] .


# Analyzing Toxicity in Deep Conversations: A Reddit Case Study

Vigneshwaran Shankaran
GESIS - Leibniz Institute for the Social Sciences
<EMAIL_ADDRESS>
Rajesh Sharma
University of Tartu
<EMAIL_ADDRESS>

###### Abstract

Online social media has become increasingly popular in recent years due to its
ease of access and ability to connect with others. One of social media’s main
draws is its anonymity, allowing users to share their thoughts and opinions
without fear of judgment or retribution. This anonymity has also made social
media prone to harmful content, which requires moderation to ensure
responsible and productive use. Several methods using artificial intelligence
have been employed to detect harmful content. However, conversation and
contextual analysis of hate speech are still understudied. Most promising
works only analyze a single text at a time rather than the conversation
supporting it. In this work, we employ a tree-based approach to understand how
users behave concerning toxicity in public conversation settings. To this end,
we collect both the posts and the comment sections of the top 100 posts from 8
Reddit communities that allow profanity, totaling over 1 million responses. We
find that toxic comments increase the likelihood of subsequent toxic comments
being produced in online conversations. Our analysis also shows that immediate
context plays a vital role in shaping a response rather than the original
post. We also study the effect of consensual profanity and observe overlapping
similarities with non-consensual profanity in terms of user behavior and
patterns.
Disclaimer This paper contains examples of explicit language that may be
disturbing to some readers.

Analyzing Toxicity in Deep Conversations: A Reddit Case Study

Vigneshwaran Shankaran GESIS - Leibniz Institute for the Social Sciences
<EMAIL_ADDRESS>Rajesh Sharma University of Tartu
<EMAIL_ADDRESS>

##  1 Introduction

In recent years, online social media platforms have become ubiquitous in our
daily lives. These platforms allow users to connect and communicate with
others in real-time. However, the nature of these interactions can vary
significantly based on their visibility. Private conversations that take place
through online communication channels, such as direct and group messaging,
have little effect on society as a whole since they do not influence or
challenge the dominant narratives, values, or expectations of society. But
conversations occurring in a public setting, such as on social media platforms
where users can participate, must adhere to the platform’s rules and the
general societal norms. However, on the downside, as online social media
provide users with a degree of anonymity, this often leads to the emergence of
hate speech on the platforms Mondal et al. (2017). Therefore, it is important
to moderate content on social media because it helps ensure that the platform
remains safe and respectful for all users. Without moderation, social media
can become a breeding ground for spam, hate speech, and other forms of harmful
or offensive content. Subsequently, this can create a negative user experience
and make the platform less enjoyable for everyone.

Given the vast quantity of content, it is difficult to manually examine and
identify every instance of hate speech on social media. Consequently, several
studies have investigated how to identify hate speech using artificial
intelligence. Existing automatic moderation systems use texts sourced from
various social media platforms and online encyclopedias manually labeled by
human annotators Waseem and Hovy (2016); Davidson et al. (2017); Wulczyn et
al. (2017). The aforementioned studies focus on texts as individual sentences
and disregard the notion of conversations that occur in the form of responses
to posted content on online social media platforms. Some social media
platforms, like Instagram, Youtube, etc., prohibit users from posting nested
responses and limit them to a single level. Whereas other platforms like
Twitter and Reddit facilitate deeper conversations by permitting nested
responses, allowing for discussions similar to real life. Moreover, it is
worth noting that certain online communities exhibit distinct norms that may
be more permissive or even toxic compared to others. In such cases, new users
often find themselves adapting to these established norms Rajadesingan et al.
(2020). Figure 1 is an example of an online conversation sourced from Reddit
where the responses to the post are ordered in a hierarchical structure.

This paper focuses on analyzing deep conversations by taking a holistic view
of a post and its responses. Specifically, this study centers on the
identification of toxic language and patterns within a given post and its
subsequent responses. To achieve this, we analyze a dataset consisting of 800
Reddit posts that have garnered over 1 million responses. These posts are
sourced from eight distinct subreddits known for hosting lengthy discussions.
We use Reddit since a response to a comment or post expresses agreement or
disagreement with the original text; it can be viewed as a public
conversation. The main contribution of this paper is to find how toxicity
disseminates in public conversations on social media platforms that support
deep conversations.

Figure 1: Excerpt from a Reddit post. Nodes with red borders signify toxic
content as classified by the ToxiGen classifier

Our analysis investigates the following research questions:

  * •

RQ1: Are toxic comments more likely to generate toxic conversations than non-
toxic ones?
We find a significant correlation between the toxicity of response and the
subsequent toxicity it attracts in terms of responses. See Section 5

  * •

RQ2: Does the toxicity of a response depend on the toxic levels of responses
made before it?
According to our analysis, the response immediately preceding a given response
has a greater impact on its toxicity compared to the previous responses. See
Section 6

  * •

RQ3: How long does a conversation retain its toxic nature?
We observe that toxicity in a conversation generally diminishes within the
initial two to three levels of responses and does not endure over time. See
Section 7

  * •

RQ4: What effect does toxicity have on user retention in a conversation?
We observe that users exhibit a bimodal pattern in terms of toxicity levels,
providing an interesting insight into the dynamics of neutral responses in
public conversations. See Section 8

  * •

RQ5: Does the toxicity norms of consensual groups differ from non-consensual
groups?
We find overlapping similarities between both groups, suggesting identical
toxicity norms. See Section 9

The remainder of the paper is structured as follows. In Section 2, we present
relevant works. We define our conversational modeling approach in Section 3.
In Section 4, we discuss the dataset, then in Sections 5 through 9, we address
the research questions and in Section 10, we provide concluding remarks.

##  2 Related Work

In this section, we discuss existing literature on topics pertinent to our
research. We explore two distinct yet interconnected domains relating to our
work: toxicity detection and user behavior analysis.

###  2.1 Toxicity Detection

The early efforts to address abusive language, such as Yin et al. (2009),
employed supervised classification with n-gram, manual regular expression
patterns, and contextual characteristics based on prior phrases’ abusiveness.
Diverse datasets on topics like racism, feminism, and misogyny were gathered
from online platforms like Twitter and Reddit Waseem (2016); Founta et al.
(2018); Waseem and Hovy (2016). Additionally, datasets, including examples of
personal attacks and toxic messages from Wikipedia’s Talk pages, were used to
analyze factors like hate speech, toxicity, and community sentiment on these
platforms Wulczyn et al. (2017). However, models trained on these datasets
struggled with implicit hate, leading ElSherief et al. (2021) to source tweets
from extremist groups for labeled implicit toxicity.

Automatic hate speech detection’s inherent complexity poses challenges, as
studies highlight difficulties arising from language variations and biases
toward certain identity groups in automated systems Sap et al. (2019); Nobata
et al. (2016); Davidson et al. (2019); Xia et al. (2020). Researchers
addressed bias concerns, introducing metrics specific to identity performance
evaluation Dixon et al. (2018); Borkan et al. (2019). Approaches like
functional test suites Röttger et al. (2021), human-in-the-loop dataset
generation Vidgen et al. (2021), and multi-task learning Vaidya et al. (2020)
aimed to enhance model robustness. Additionally, Barikeri et al. (2021) used
embedding projections to mitigate unintended bias. While previous works
focused on single textual entities, Ghosh et al. (2018) analyzed sarcasm using
LSTM and attention models, and contextual analysis in fake news detection
incorporated hyperbolic geometric representations and Fourier transformations
Grover et al. (2022). An extension of this contextual approach was adapted for
implicit hate speech detection Ghosh et al. (2023).

###  2.2 User Behavior Analysis

While studies have improved toxicity detection models, a different line of
work has analyzed social media platforms to unearth user behavior and
information dissemination patterns in human communications in cyberspace.
Aumayr et al. (2011) pioneered methods for reconstructing reply behaviors in
threads, employing a classification approach that integrates both content and
non-content features. Leavitt and Clark (2014) delved into the production of
news on a hurricane-related subreddit, investigating categories of news likely
to receive more upvotes. Notably, followers of influencers disseminating
deceptive content on social media have been observed to employ uncivil
language and become more effectively polarized Guldemond et al. (2022).

In addition to broader analytics on social media platforms, there has been a
noteworthy surge in literature specifically examining toxicity within these
platforms. A study on Twitter user data by Yousefi et al. (2023) highlighted
how various content types and contexts can impact the spread of toxicity
within online communities. Furthermore, a cross-platform study indicated that
Reddit tends to exhibit higher toxicity levels than other platforms, such as
Twitter and Parler Noor et al. (2023), thereby influencing our choice of
Reddit as a data source. An insightful examination of Reddit users revealed
that usernames serve as useful predictors for author profiling, as users with
toxic usernames tend to generate more toxic content than others Urbaniak et
al. (2022).

The works closest to ours involve studies that have analyzed more than one
post at a time with respect to toxicity. For example, Bakshy et al. (2012)
delved into information diffusion on Facebook, seeking insights into the
factors influencing post reach. Aleksandric et al. (2022) conducted a
longitudinal study on Twitter, uncovering behavioral reactions of toxicity
victims, including avoidance, revenge, countermeasures, and negotiation. Kumar
et al. (2010) explored the dynamics of online conversations, analyzing
patterns in human exchanges on social media, scrutinizing thread depth, and
correlating their findings with Heap’s law Heaps (1978). While previous works
have investigated user interaction behavior in public conversations, such as
continuity Kim et al. (2023) and new & re-entry Wang et al. (2022), our
approach distinguishes itself by analyzing deep public conversations using a
tree-based conversation modeling approach. This methodology enables us to
capture the intricate interplay of context and toxicity without imposing
limitations on contextual depth, providing valuable insights into how toxicity
shapes the buildup of conversations in a public setting.

##  3 Conversation Modeling

By and large, in online social media or content-sharing platforms, users post
content and receive responses in the form of likes, replies, or comments. In
particular, responses in the form of comments facilitate the generation of
conversations in a public setting which can be either deep or shallow,
depending on the platform. We refer to comments as responses henceforth in
this paper. Platforms like Youtube, Instagram, and Facebook allow for shallow
conversations with little engagement since only a single level of responses is
permitted. However, Twitter and Reddit promote a higher level of engagement
owing to their support for deep conversations. Due to the inherent
conversational structure of deep conversations, they can be represented as a
generic tree model where the post and responses are represented as nodes.
Figure 1 depicts a deep conversation in which the post or content by itself is
the root node, and the remaining nodes are responses. The responses of each
individual node are represented as child nodes. Note that every conversation
has the possibility of containing toxic responses due to the acceptance of
profanity in the chosen subreddits. To proceed further with our analysis, we
define the following:

  * •

Opinions : Opinions can be of any reply with relation to the node’s content.
In other words, an opinion does not necessarily need to be supportive or
against the node’s content as it could be neutral, too. Opinion is determined
by the number of children nodes of a response. More responses (children)
correspond to more opinions for any given response or post.

  * •

Engagement : This represents a node’s (response) maximum depth. A deeper
branch of response indicates that there is a higher level of interest and
participation in the discussion, encouraging users to continue contributing
their thoughts and opinions on the topic. As a result, a node with a greater
depth is often seen as a sign of successful engagement and active
participation.

  * •

Toxic Accumulation : Every node comprising of text is treated as a single
entity and a toxicity score is computed based on it. The toxic accumulation
score is computed by averaging the toxicity scores of all the nodes that
branch from the given node. This indicates the level of toxicity produced by a
node. A higher score indicates that the response contributed to the emergence
of toxic conversations. Toxic accumulation can be defined by the following
equations:

$TA_{SubTree}(Node_{i})=\sum_{j}^{\\#Children}TA({Node_{i}}_{j})$

(1)

$TA(Node_{i})=\frac{Toxicity(Node_{i})+TA_{SubTree}(Node_{i})}{\\#Children+1}$

(2)

In Equation 1, $TA_{SubTree}(Node_{i})$ denotes the contribution of toxicity
by subtree formed by $Node_{i}$. It must be noted that Equations 1 and 2 are
recursive and complement each other. Equation 2 computes toxic accumulation of
$Node_{i}$. Therefore, $TA({Node_{i}}_{j})$ in Equation 1 refers to the
summation of toxic accumulation of all the children $j$ of $Node_{i}$.

These definitions help us better understand user participation in the presence
of toxic comments by establishing the relationship between engagement and
opinions as well as toxicity and toxic accumulation.

##  4 Dataset

We use Reddit posts for analysis as the platform facilitates deep
conversations and provides a large quantity of data with diverse discussions,
allowing us to analyze the nature of deep conversations on the platform
thoroughly. According to Reddit’s algorithm111https://github.com/reddit-
archive/reddit/blob/master/r2/r2/lib/db/_sorts.pyx, top posts are categorized
by upvotes and user participation among other factors and therefore are prone
to generate deep conversation, which suits our purpose. The top 100 posts as
of October 12th, 2022 were chosen from the following eight subreddits:
r/atheism, r/confidentlyincorrect, r/Fuckthealtright, r/antifeminists,
r/facepalm, r/4chan, r/tifu and r/RoastMe. As the titles of the subreddits
imply, one may assume that we chose them for the controversial aspects they
might encourage, however, these subreddits were chosen for free discourse,
debate, and communal acceptance of profanity in the respective subreddits. We
extracted the posts and comments while preserving the hierarchy of the
conversation. Each comment’s text was collected along with its metadata,
including the score, anonymized author ID, comment ID and time stamp, and
stored as individual JSON files222https://github.com/s-vigneshwaran/Reddit-
Case-Study-Data.

Figure 2: Distribution of the number of responses to posts

Reddit allows multiple forms of content as the body of a post, like HTML,
image, audio, gif, etc. A manual examination of 10 randomly selected posts
from each subreddit revealed that many posts included pictures with text in
some way, be it a quote, a screenshot of a news article, or a Twitter post,
etc. We extracted text from the images of such posts and included it in the
main body of the post manually because the conversation generated has a direct
effect on the post’s content. Upon inspection of the entire data (after
extracting texts from images), we found that the posts had a median of 52
words, and the responses had a median of 16 words. Figure 2 shows the
distribution of the number of responses to posts signifying that majority of
post attracts fewer than 2000 responses. It must be noted that we exclude
posts from r/RoastMe and use only 700 posts for Sections 5 to 8 since they are
targeted towards analyzing non-consensual profanity. The remaining 100 posts
from r/RoastMe has been utilized in Section 9 to study consensual profanity
patterns. For a detailed breakup, refer to Table 1.

Subreddits | Comments
---|---
r/atheism | 116,927
r/confidentlyincorrect | 189,124
r/Fuckthealtright | 14,534
r/antifeminists | 9,257
r/facepalm | 378,051
r/4chan | 39,957
r/tifu | 203,652
r/RoastMe | 152,008
Total | 1,103,500

Table 1: Dataset statistic

Ethical Consideration Our study does not use any user information in our
analysis, and the presented results are all shown at an aggregated level,
which cannot be traced back to a specific user in any way. Moreover, all the
user information is anonymized by obscuring their unique IDs to protect their
privacy.

### Toxicity Analysis

We compute the toxicity of each node and then determine the toxic accumulation
for each node. To evaluate the toxicity of the responses, we utilized an off-
the-shelf RoBERTa Liu et al. (2019) trained on the ToxiGen dataset333Publicly
available at https://huggingface.co/tomh/toxigen_robertaHartvigsen et al.
(2022). ToxiGen is a large, well-balanced, adversarially generated corpus of
implicit toxic language that can identify subtle toxic content and has the
ability to differentiate between different targeted identities, thereby
reducing bias. However, to ensure ToxiGen’s suitability for our purpose, we
tested its efficacy with Google’s Perspective
API444https://perspectiveapi.com/research/. We sampled 70 posts totaling
103,967 responses from the dataset and computed toxicity scores using ToxiGen
and Perspective API. Subsequently, in order to assess the extent of agreement
between the aforementioned models, we calculated Cohen’s $\kappa$, resulting
in a value of 0.6, indicative of a moderate level of agreement. On inspection
of the disagreements, interestingly, we find that ToxiGen performs extremely
well in implicit hate; for example, for the text "Allanus Whoreisette",
Perspective API score was 0.02 whereas ToxiGen predicted a score of 0.98. As a
result of increased effectiveness for implicit hate, every node of a post,
including the root node, was assigned a "toxicity score" computed using
ToxiGen and logged for further analysis. Using this score as a starting point,
we calculated the proposed toxic accumulation (Section 3) of all of the
responses in our corpus. The toxicity and toxic accumulation scores drive our
analysis further.

##  5 Relationship between toxicity and toxic accumulation

In this section, we investigate “RQ1: Are toxic comments more likely to
generate toxic conversations than non-toxic ones?”. The general connotation is
that a toxic comment will cause the target to withdraw from the conversation
Kang et al. (2016). This suggests that, in the context of a public
conversation, a toxic response has the potential to alter the conversation’s
content and direction as users may or may not react to toxic responses. Given
toxic accumulation’s ability to aggregate toxicity scores across the entire
sub-branch, it serves as a dependable measure for estimating the toxicity
produced by a response, thus making it the preferred approach for addressing
this research question.

###  5.1 Methodology

To answer RQ1, we explored correlation statistic. Firstly, we collect all
responses ignoring the hierarchy from the dataset, along with its toxicity and
toxic accumulation scores. Then we calculate the Pearson correlation
coefficient between the two defined scores across the entire dataset. If the
response is toxic, this statistical metric can be used to ascertain whether or
not the sub-branch is likely to be toxic as a result of the toxic accumulation
of the node. Since the toxicity and toxic accumulation will be the same in the
terminal responses (leaf nodes) we excluded them to avoid confounding our
results.

###  5.2 Results

The correlation between the toxicity and toxic accumulation of the comments is
presented in Table 2. The average correlation score is 0.631 $\pm$ 0.013,
indicating a moderate degree of association. The correlation is statistically
significant (P-Value $<$ 0.01). The top two subreddits concerning this metric
(r/tifu, and r/facepalm) are in no way controversial, unlike subreddits that
are strongly centered around topics such as r/atheism, r/antifeminsts, and
r/Fuckthealtright denoting the presence of variety in the data. By including
the leaf nodes, the correlation turns out to be 0.9 for the entire dataset,
demonstrating that leaf nodes do, in fact confound the results.

Subreddit | Correlation
---|---
r/tifu | 0.649
r/facepalm | 0.643
Entire Dataset | 0.641
r/athesim | 0.638
r/confidentlyincorrect | 0.628
r/Fuckthealtright | 0.627
r/antifeminists | 0.613
r/4chan | 0.610

Table 2: Correlation between toxicity and toxic accumulation

In light of these results, it becomes evident that toxic comments have a
significant influence on the overall tone and trajectory of online discussions
as they possess the potential to escalate and perpetuate toxicity within
conversations, leading to more negativity and hostility on social media
platforms.

##  6 Contextual Analysis

One of the main benefits of preserving the structure of a public conversation
is the ability to analyze the influence of previous responses on the target
response effectively. This allows us to answer the question "RQ2: Does the
toxicity of a response depend on the responses made before it?" This analysis
is focused solely on the toxic nature of responses and does not consider the
content or meaning of previous responses i.e, we give importance to toxic
context (score) over linguistic context (text). Figure 1 shows an example of
an excerpt from a post to highlight the importance of context in terms of
toxicity. Note that toxic nodes yield more toxic responses in the given
example.

###  6.1 Methodology

We examine whether previous responses impact the toxicity of the target
response. To achieve this, we collect toxicity scores of up to 5 previous
comment levels of every response in the dataset. For example, in Figure 1, for
the response labeled 2.1, the immediate predecessor is labeled 2, and the
extended predecessor is labeled 0 and is termed level 2 analysis since we use
only two levels of context, i.e, we understand how the toxicity of 0 and 2
affect 2.1. Analyzing the relationship between the toxicity of the predecessor
and the toxicity of the extended predecessor demonstrates its effect on a
response. Similarly, levels 3, 4, and 5 follow the same pattern, but the
context becomes more in-depth. We modeled four simple linear regression
models: the target variable is a response’s toxicity, and the dependent
variables are the responses’ preceding comments.

$T=\beta_{0}+\sum_{i}^{\\#Levels}\beta_{i}T_{Level_{i}}$

(3)

Equation 3 represents a simple linear regression model, $Level_{i}$ refers to
the previous levels’ toxicity scores. $i$ takes in the levels for which the
target variable is modeled. For instance, a level $2$ analysis would take $i$
as $1$ and $2$ where $1$ is parent and $2$ is a grandparent.

###  6.2 Results

Table 3 presents the beta coefficients of the models, capturing valuable
insights. It is evident that the immediate predecessor holds a greater
influence on the toxicity of the response compared to previous levels, while
the impact of predecessors’ toxicity remains minimal. Despite the low
significance, we can observe a consistent downward trend in the impact levels,
suggesting that the previous response plays a role in shaping the toxicity of
a response within a public conversation. Similar results were obtained when
analyzing individual subreddits utilizing the same methodology. Due to strong
overlapping similarity, we don’t report the subreddit-wise beta coefficients
in the paper.

Levels | n-1 | n-2 | n-3 | n-4 | n-5
---|---|---|---|---|---
2 | 0.1274 | 0.0679 |  |  |
3 | 0.1236 | 0.0898 | 0.0334 |  |
4 | 0.1155 | 0.0925 | 0.0385 | 0.0375 |
5 | 0.1138 | 0.0974 | 0.0316 | 0.0460 | 0.0248

Table 3: Linear significance of previous levels’ toxicity on comment’s
toxicity

##  7 Temporal Behavioral Change

Figure 3: Total number of conversational branches

This section examines the characteristics of public conversations in terms of
change with respect to time. Here, time corresponds to the depth of a
conversation’s branches. We examine the duration of toxic prevalence in a
conversation, answering the question "RQ3: How long does a conversation retain
its toxic nature?". We find that toxic conversations are generally lesser in
prevalence than non-toxic conversations.

###  7.1 Methodology

Each branch, characterized by its toxicity scores, forms a sequence of numbers
where each value depends on the preceding levels. This configuration allows us
to treat the data as time series, offering insights into the evolving toxicity
levels as the conversation progresses. The distribution of conversational
branches concerning comment levels in the dataset is illustrated in Figure 3.
A consistent decline in the number of branches is evident beyond ten levels,
with only 26,958 out of 758,519 branches extending beyond this threshold. To
address RQ3, we establish a ten-level threshold for analysis. Subsequently,
the mean toxicity scores and accumulated toxicity scores for selected branches
are computed and visually examined to discern any trends in the rate of change
concerning toxicity.

(a)

(b)

(c)

(d)

Figure 4: Toxic trends of conversations. X-axis denotes the comment level and
Y-axis denotes the mean toxic metric

###  7.2 Results

Figure 4 illustrates the trends of toxicity and toxic accumulation over time.
Subfigures 4(a) and 4(c) communicate the trends irrespective of the nature of
the posts, whereas, Subfigures 4(b) and 4(d) describe the toxic trends for
posts whose toxic threshold cross 0.5. It can be observed that after the first
three levels, the toxicity tends to level off irrespective of the comment
being toxic or not toxic (toxicity score $>$ 0.5), signifying that
conversations in public setting generally does not hold shape for long. This
also resonates with the harmful effects of toxicity, as toxic comments have
the effect of causing users to lose interest in the content and move on from
the conversation Schmid et al. (2022). Interestingly, the subreddit r/4chan is
slightly distinct from others because, in contrast to other subreddits, the
toxic levels are generally high, decrease steadily, and never fluctuate. It is
likely the result of reposting content onto this subreddit from the platform
4chan, which is known for its notorious toxic levels due to its anonymity.

##  8 User participation with toxic comments

Figure 5: Opinions generated on different levels of toxicity

This section undertakes an investigation into the repercussions of toxicity on
user retention in the context of public conversations. To achieve this
objective, we employ the definitions of engagement and opinion as delineated
in Section 3 to ascertain the influence of these variables on user
participation. Through a very simple analysis of the intricate interplay among
toxicity and opinions, we derive rudimentary insights pertaining to the
ramifications of toxicity on user retention within the conversational realm,
thus addressing the research question posited as "RQ4: What effect does
toxicity have on user retention within a conversation?".

###  8.1 Methodology

To begin our analysis, we categorize the spectrum of toxicity into ten groups
with a step range of 0.1, with group 1 representing non-toxic and group 10
representing highly toxic, enabling a more granular examination of the
toxicity levels under consideration. Following this, we take the sum of the
number of opinions garnered by all these ten groups in the dataset. Due to the
significant disparity in the dataset regarding toxicity levels $<$ 0.1,
indicating a substantial portion of non-toxic responses on social media, the
outcome could be heavily biased. As a result, we made the decision to remove
all data points associated with group 1.

###  8.2 Results

Figure 5 illustrates the total number of responses categorized by toxicity
range. In the plot, one can observe the presence of bimodal phenomena. This
phenomenon indicates that the number of opinions increases when the text is
either too toxic (toxic score $>$ 0.8) or normal. This further adds to the
reasoning that people prefer to continue responding to non-toxic texts that
adhere to positive and respectful communication norms rather than engaging
with toxic content. However, we also observe a counter-intuitive result that
the number of responses increases as the toxicity increases. Upon further
inspection of parent-child response pairs, we find that out of 190,929 toxic
parent responses 27.9% children responses are toxic whereas out of 911,771
non-toxic parent responses only 19.3% children responses are toxic signifying
that toxic responses has better chance of inviting more toxic responses.

##  9 Effect of Consensual Profanity

Abusive language is not just used to attack individuals; it may also be used
to frame profane information in civil discourse Jordan (2020). We analyze
consensual toxic comments and compare them with unsolicited toxic and hateful
content which are not consensual to answer "RQ5: Does the toxicity norms of
consensual groups differ from non-consensual groups?". To this end, we scrape
the responses of the top 100 posts from r/RoastMe, totaling 152,008 responses
from the temporal distribution as our dataset. We chose r/RoastMe since it is
a community where users post photos of themselves asking to be roasted (severe
criticism) by other members of the community. We performed the same set of
analyses mentioned in the previous sections to understand if consensual
toxicity patterns resemble the general toxic norms in public discussions.

The correlation between toxicity and toxic accumulation for the responses
belonging to this subreddit is 0.66, which is slightly higher than that of the
non-consensual subreddits (Refer to Table 2). However, the difference is a
minuscule +0.03 from the mean correlation. The contextual analysis on
r/RoastMe resulted in the same significance pattern of the beta coefficients
of the predecessors signaling that the immediate predecessor’s toxicity
impacts a node more than extended predecessors for the non-consensual
communities as well. The mean scores of toxicity and accumulated toxicity
resulted in the same trend overlapping with the patterns observed in Figure 5.
The total number of opinions generated 141,757 responses for toxicity levels
$<$ 0.3 & $>$ 0.8 and 10,251 for toxicity in the range [0.3, 0.8]. The results
help us in establishing the similarity between the toxic norms of consensual
and non-consensual communities.

##  10 Conclusion

In this paper, we comprehensively analyze toxicity in public conversations on
online social media platforms. We examined 800 Reddit posts with over 1
million responses using a tree hierarchy structure. We determined the toxicity
of each response using a RoBERTa model that was pre-trained on an
adversarially generated dataset specifically designed to combat subtle
toxicity. Our findings suggest that toxic comments have multiple negative
effects on the continuation of a public conversation. For instance, we
observed that the presence of toxicity in a conversation increases the
likelihood of future toxic comments and discourages users from continuing to
participate. An observation from our study is that users tend to interact less
with responses when the content is neutral when compared to toxic or non-toxic
content. Interestingly, consensual and non-consensual toxic communities follow
the same toxic norm patterns.

Limitations In order to ensure that our dataset is as unbiased as possible, we
took extra care in selecting it. Despite our efforts, however, there may still
be some degree of bias present in the dataset.

## Ethical Consideration

Our research does not use any user information in our analysis, and the
presented results are all shown at an aggregated level, which cannot be traced
back to a specific user in any way. Moreover, all the user information is
anonymized by removing or obscuring their unique IDs to protect their privacy.

## References

  * Aleksandric et al. (2022) Ana Aleksandric, Sayak Saha Roy, and Shirin Nilizadeh. 2022.  Twitter users’ behavioral response to toxic replies.  _arXiv preprint arXiv:2210.13420_.
  * Aumayr et al. (2011) Erik Aumayr, Jeffrey Chan, and Conor Hayes. 2011.  Reconstruction of threaded conversations in online discussion forums.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 5, pages 26–33.
  * Bakshy et al. (2012) Eytan Bakshy, Itamar Rosenn, Cameron Marlow, and Lada Adamic. 2012.  The role of social networks in information diffusion.  In _Proceedings of the 21st international conference on World Wide Web_ , pages 519–528.
  * Barikeri et al. (2021) Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021.  RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1941–1955, Online. Association for Computational Linguistics.
  * Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019.  Nuanced metrics for measuring unintended bias with real data for text classification.  In _Companion proceedings of the 2019 world wide web conference_ , pages 491–500.
  * Davidson et al. (2019) Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019.  Racial bias in hate speech and abusive language detection datasets.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 25–35, Florence, Italy. Association for Computational Linguistics.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the international AAAI conference on web and social media_ , volume 11, pages 512–515.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 67–73.
  * ElSherief et al. (2021) Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021.  Latent hatred: A benchmark for understanding implicit hate speech.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 345–363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Founta et al. (2018) Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of twitter abusive behavior.  In _Twelfth International AAAI Conference on Web and Social Media_.
  * Ghosh et al. (2018) Debanjan Ghosh, Alexander R. Fabbri, and Smaranda Muresan. 2018.  Sarcasm analysis using conversation context.  _Computational Linguistics_ , 44(4):755–792.
  * Ghosh et al. (2023) Sreyan Ghosh, Manan Suri, Purva Chiniya, Utkarsh Tyagi, Sonal Kumar, and Dinesh Manocha. 2023.  Cosyn: Detecting implicit hate speech in online conversations using a context synergized hyperbolic network.  _Preprint_ , arXiv:2303.03387.
  * Grover et al. (2022) Karish Grover, SM Angara, Md Shad Akhtar, and Tanmoy Chakraborty. 2022.  Public wisdom matters! discourse-aware hyperbolic fourier co-attention for social text classification.  _Advances in Neural Information Processing Systems_ , 35:9417–9431.
  * Guldemond et al. (2022) Puck Guldemond, Andreu Casas Salleras, and Mariken van der Velden. 2022.  Fueling toxicity? studying deceitful opinion leaders and behavioral changes of their followers.  _Politics and Governance_ , 10(4):336–348.
  * Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022.  ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3309–3326, Dublin, Ireland. Association for Computational Linguistics.
  * Heaps (1978) Harold Stanley Heaps. 1978.  _Information retrieval, computational and theoretical aspects_.  Academic Press.
  * Jordan (2020) John W Jordan. 2020.  Profanity from the heart as exceptional civic rhetoric.  _Quarterly Journal of Speech_ , 106(2):111–132.
  * Kang et al. (2016) Ruogu Kang, Laura Dabbish, and Katherine Sutton. 2016.  Strangers on your phone: Why people use anonymous communication applications.  In _Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing_, pages 359–370.
  * Kim et al. (2023) Jinhyeon Kim, Jinyoung Han, and Daejin Choi. 2023.  Predicting continuity of online conversations on reddit.  _Telematics and Informatics_ , 79:101965.
  * Kumar et al. (2010) Ravi Kumar, Mohammad Mahdian, and Mary McGlohon. 2010.  Dynamics of conversations.  In _Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 553–562.
  * Leavitt and Clark (2014) Alex Leavitt and Joshua A Clark. 2014.  Upvoting hurricane sandy: event-based news production processes on a social news site.  In _Proceedings of the SIGCHI conference on human factors in computing systems_ , pages 1495–1504.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mondal et al. (2017) Mainack Mondal, Leandro Araújo Silva, and Fabrício Benevenuto. 2017.  A measurement study of hate speech in social media.  In _Proceedings of the 28th ACM conference on hypertext and social media_ , pages 85–94.
  * Nobata et al. (2016) Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016.  Abusive language detection in online user content.  In _Proceedings of the 25th international conference on world wide web_ , pages 145–153.
  * Noor et al. (2023) Nahiyan Bin Noor, Niloofar Yousefi, Billy Spann, and Dr Nidhi Agarwal. 2023.  Comparing toxicity across social media platforms for covid-19 discourse.  _ArXiv_ , abs/2302.14270.
  * Rajadesingan et al. (2020) Ashwin Rajadesingan, Paul Resnick, and Ceren Budak. 2020.  Quick, community-specific learning: How distinctive toxicity norms are maintained in political subreddits.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 14, pages 557–568.
  * Röttger et al. (2021) Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021.  HateCheck: Functional tests for hate speech detection models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 41–58, Online. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th annual meeting of the association for computational linguistics_ , pages 1668–1678.
  * Schmid et al. (2022) Ursula Kristin Schmid, Anna Sophie Kümpel, and Diana Rieger. 2022.  How social media users perceive different forms of online hate speech: A qualitative multi-method study.  _New Media & Society_, page 14614448221091185.
  * Urbaniak et al. (2022) Rafal Urbaniak, Patrycja Tempska, Maria Dowgiałło, Michał Ptaszyński, Marcin Fortuna, Michał Marcińczuk, Jan Piesiewicz, Gniewosz Leliwa, Kamil Soliwoda, Ida Dziublewska, et al. 2022.  Namespotting: Username toxicity and actual toxic behavior on reddit.  _Computers in Human Behavior_ , 136:107371.
  * Vaidya et al. (2020) Ameya Vaidya, Feng Mai, and Yue Ning. 2020.  Empirical analysis of multi-task learning for reducing identity bias in toxic comment detection.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 14, pages 683–693.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1667–1682, Online. Association for Computational Linguistics.
  * Wang et al. (2022) Lingzhi Wang, Jing Li, Xingshan Zeng, and Kam-Fai Wong. 2022.  Successful new-entry prediction for multi-party online conversations via latent topics and discourse modeling.  In _Proceedings of the ACM Web Conference 2022_ , pages 1663–1672.
  * Waseem (2016) Zeerak Waseem. 2016.  Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter.  In _Proceedings of the first workshop on NLP and computational social science_ , pages 138–142.
  * Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016.  Hateful symbols or hateful people? predictive features for hate speech detection on Twitter.  In _Proceedings of the NAACL Student Research Workshop_ , pages 88–93, San Diego, California. Association for Computational Linguistics.
  * Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017.  Ex machina: Personal attacks seen at scale.  In _Proceedings of the 26th international conference on world wide web_ , pages 1391–1399.
  * Xia et al. (2020) Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020.  Demoting racial bias in hate speech detection.  In _Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media_ , pages 7–14, Online. Association for Computational Linguistics.
  * Yin et al. (2009) Dawei Yin, Zhenzhen Xue, Liangjie Hong, Brian D Davison, April Kontostathis, and Lynne Edwards. 2009.  Detection of harassment on web 2.0.  _Proceedings of the Content Analysis in the WEB_ , 2:1–7.
  * Yousefi et al. (2023) Niloofar Yousefi, Nahiyan Bin Noor, Billy Spann, and Dr Nidhi Agarwal. 2023.  Towards developing a measure to assess contagiousness of toxic tweets.


empirical risk minimization, robust learning, vc dimension, tolerant learning