# Multi-pixel Photon Counter for operating the tabletop cosmic-ray detector
under loosely controlled conditions

N. H. Duy Thanh<EMAIL_ADDRESS>Institute for Interdisciplinary
Research in Science and Education,
ICISE, Quy Nhon, Vietnam. N. V. Chi Lan  Institute for Interdisciplinary
Research in Science and Education,
ICISE, Quy Nhon, Vietnam. S. Cao<EMAIL_ADDRESS>Institute for
Interdisciplinary Research in Science and Education,
ICISE, Quy Nhon, Vietnam. High Energy Accelerator Research Organization (KEK),
Tsukuba, Ibaraki, Japan.
T. V. Ngoc  Institute for Interdisciplinary Research in Science and Education,
ICISE, Quy Nhon, Vietnam. Graduate University of Science and Technology,
Vietnam Academy of Science and Technology, Hanoi, Viet Nam. N. Khoa  Albion
College, Albion, MI, United States N. T. Hong Van  Institute of Physics,
Vietnam Academy of Science and Technology, Hanoi, Vietnam. P. T. Quyen
Institute for Interdisciplinary Research in Science and Education,
ICISE, Quy Nhon, Vietnam. Graduate University of Science and Technology,
Vietnam Academy of Science and Technology, Hanoi, Viet Nam.

###### Abstract

Multi-Pixel Photon Counter (MPPC) has been recently emerged and realized as a
great type of Silicon Photomultiplier to replace or compensate for the
conventional vacuum-based Photomultiplier tubes. MPPC provides many striking
features such as high electrical gain, outstanding photon detection
efficiency, fast timing response, immunity to the magnetic field, low-voltage
operation, compactness, portability, and cost-effectiveness. The report
introduces and examines the electrical and optical characteristics of the MPPC
under loosely controlled environmental conditions. Also, we report a
measurement of the light yield captured by the MPPC when the cosmic ray passes
through the plastic scintillator, demonstrating that such setup is suitable to
build a simple, cost-effective tabletop cosmic-ray detector for educational
and research purposes.

##  I Introduction

###  I.1 Silicon photodiode, avalanche breakdown, and Geiger mode

A photodiode, as illustrated in the left of Fig. 1, is made from a silicon
doped p-n junction which establishes a depletion region of mobile carriers in
the thermal equilibrium Neamen (2011). When a photon is incident on silicon, a
pair of electrons and holes are created via the photoelectric effect in the
depletion region. When a reverse-biased voltage is applied to the silicon
photodiode, generating a high electric field across the depletion region, the
produced charge carriers experience an increase in the kinetic energy and
collide with the lattice atoms, eventually losing their energy as thermal
phonons. When the applied voltage is high enough, the carriers have sufficient
power to ionize the lattice atoms, inducing additional electron-hole pairs and
resulting in a chain of ionization called _avalanche breakdown_ or _avalanche
multiplication_. Thus, an absorbed photon can trigger a macroscopic electrical
current, allowing us to distinctly detect the photon’s incident. A photodiode
called _Avalanche PhotoDiode_ (APD) is designed to operate in such conditions.
The charge collected from the avalanche multiplication is expressed in terms
of the electron charge unit and called electrical gain.

Figure 1: Operational principle of a P-N junction and classification of
photodiode (PD), avalanche photodiode (APD) and SiPM operating in the Geiger-
mode avalance.

When the operating voltage is lower than the APD breakdown voltage, a turning
point where the silicon suddenly becomes conductive subjecting to the external
electric field, the electrical gain has a linear response to the applied
voltage and typically less than one thousand. When the reverse bias voltage
exceeds the APD breakdown voltage, the electrical gain increases drastically.
The APD becomes a binary response, such that the output charge is independent
of the intensity of incident photons. Silicon Photomultiplier (SiPM), denoted
for the APD operated in this so-called Geiger mode, has an electrical gain of
$~{}10^{6}$. Classification of the photodiode, APD, and SiPM basing on the
amplitude of the reverse bias voltage is illustrated on the right of Fig. 1.

###  I.2 Multi-Pixel Photon Counter

Multi-Pixel Photon Counter (MPPC) Ghassemi  _et al._ (2018) is one of the most
used SiPM, developed by Hamamatsu, Japan. It is fabricated on a monolithic
silicon crystal, with a composite-metal quenching resistor around each APD,
forming a so-called pixel. The multiple pixels are arranged in two dimensions,
connected in parallel with a common reverse bias voltage, and read by a united
output. An equivalent circuit of MPPC and a close-up view of a single MPPC are
shown in Fig. 2. Since each pixel produces the same electrical current per
incident photon, the MPPC signal summing the currents from the fired pixels is
practically equivalent to the number of incident photons, providing MPPC
excellent capability to count photons.

Figure 2: The left is an equivalent circuit of MPPC with APD pixels connected
in parallel Ghassemi  _et al._ (2018). The right is a close-up view of the
MPPC with 667 pixels manufactured on a 1.3$\times$1.3 $\text{mm}^{2}$
sensitive area Hamamatsu Photonics K.K. .

The first generation of Hamamatsu MPPCs, the S10362 series, was announced in
2007 Ohashi (2019). Six years later, the second generation of MPPCs, the
S1257x series, was released with lower afterpulse and improved photon
detection efficiency (PDE) but still relatively high crosstalk. In 2015, they
introduced a new generation of MPPCs, the S1336x series, with lower crosstalk
and afterpulse and higher fill factor. Compared to the S1257x series, this
series provides a higher PDE for a broader range of wavelengths from UV to
visible light. In 2017, a new lineup of products, the S14160/S14161 series,
dedicating to the time-of-flight application such as the Positron Emission
Tomography (PET) with excellent coincidence time resolution. Recently, two
additional lineups of products Yamamoto  _et al._ (2019) have been introduced:
(i) S14520 series designed for the Cherenkov telescopes with high PDE in the
UV region, and (ii) S14160 series with a pixel size of 10 $\mu m$ and 15 $\mu
m$ and new design of the pixel trench, achieving lower noise and crosstalk,
higher PDE, wide dynamic range, and lower power consumption.

Fig. 3 shows the electrical pulses induced by the thermal noise with two
generations of the MPPC. The signal is amplified with a NIM amplifier module
by a factor of one hundred and read out by the Tektronix TBS1104 100Mhz
bandwidth digital storage oscilloscope. The electrical pulses with a single
photoelectron (p.e) and two p.e. are visibly well distinguished for both MPPC
generations. The signal’s rising time is around a few ns and the signal’s
decay time is a few of 10 ns.

Figure 3: MPPC waveforms with the first (left) and the third (right) MPPC
generations triggered by the thermal noise. Electrical pulses with a one p.e.
and two p.e. are clearly distinct. The third MPPC generation suppresses
observably the afterpulse and crosstalk.

As studied in Ref. Seifert  _et al._ (2009), the MPPC signal can be well-
simulated with an equivalent electronic circuit implemented in the SPICE
simulation program111http://bwrcs.eecs.berkeley.edu/Classes/IcBook/SPICE/. We
have worked out to come up with a SPICE model presented on the left of Fig. 4
for the MPPC S10362-13-050C. An MPPC pixel is modeled by a capacitor Cd = 15
fF and a resistor Rd = 1 k$\Omega$. The quenching is simulated by a capacitor
of Cq = 4.3 fF and Rq = 500 k$\Omega$. The parasitic capacitance is Cg = 59
pF. The right of Fig. 4 shows the agreement between the MPPC signal recorded
by oscilloscope TBS1104 and SPICE simulation of the MPPC electrical model. The
effect of the 100 Mhz bandwidth of the oscilloscope is included in the
simulation with Rosc = 1 k$\Omega$ and Cosc = 1.59 pF.

Figure 4: The left is a simulated circuit of MPPC with 100 MHz bandwidth of
oscilloscope included. The right compares the observed MPPC waveform and the
simulation.

With a large number of excellent features and a wide selection of product
lines, MPPC is applicable in diverse fields, including, but not limited to,
extremely low-light detection in particle and nuclear physics, time-of-flight-
based distance measurement for the automotive vehicle, fluorescence and
chemiluminescence spectroscopy, biochemical sensors, single molecular
detection, positron emission tomography in the medical imaging system, quantum
computing and crytography 222see
https://www.hamamatsu.com/jp/en/product/optical-
sensors/mppc/application/index.html. In particle physics, mass production of
MPPCs was firstly used in the T2K experiment Yokoyama  _et al._ (2010) in 2008
with approximately 60,000 MPPC pieces.

##  II Characteristics of MPPC under temperature-uncontrolled conditions

In this section, we explore the electrical and optical characteristics of
MPPC. Notably, we examine these characteristics with measurements where the
environmental conditions such as temperature and humidity are not strictly
controlled. We use a commercial air conditioner to regulate the temperature
through our measurements. The room temperature is checked occasionally with an
indoor thermometer and valued at $27\pm 1^{\circ}$ C. However, the MPPC
temperature is not measured precisely and is not monitored continuously.

###  II.1 Electrical gain

Electrical gain of MPPC is defined as the number of charge carriers (electrons
or holes) produced from an incident photon on a single pixel of MPPC as
follows

$Gain=\frac{Q}{e}=\frac{C_{\text{pixel}}\times(V_{\text{op}}-V_{\text{bd}})}{e},$
(1)

where $Q$ is the output charge of avalanche multiplication; $e=1.602\times
10^{-19}$ C is the elementary charge; $C_{\text{pixel}}$ is the overall pixel
capacitance; and $V_{\text{op}}$ and $V_{\text{bd}}$ are the operating voltage
and the breakdown voltage of the MPPC respectively. The over-voltage, defined
as $\Delta V_{\text{over}}=V_{\text{op}}-V_{\text{bd}}$ , is among the most
imporant operational parameters of MPPC. The MPPC’s electrical gain is
expected to be linear with $\Delta V_{\text{over}}$ since the higher the over-
voltage is, the more available ionization energy for the charge carriers is.
From Eq. 1, with a pixel capacitor of $C_{\text{pixel}}=40~{}\text{fF}$, for
example, and an over-voltage of $\Delta V_{\text{over}}=3~{}\text{V}$, MPPC
can attain an electrical gain of $7.5\times 10^{5}$, which is at similar level
as the PMT.

At higher temperatures, the breakdown voltage increases since the lattice
vibrations become stronger. Avalanche carriers are more likely to lose kinetic
energy via scattering with the lattice atoms, reducing ionization probability.
As a result, the gain declines with the increase of temperature. Hence, to
maintain a stable operation, it is often to either keep the MPPC in a
temperature-controlled environment or employ a temperature compensation
circuit to adjust the reverse bias voltage automatically. However, in this
study, we will examine the linearity of the MPPC electrical gain when the
environmental condition for MPPC operation is not strictly controlled.

#### Measuring the electrical gain of the MPPC

To measure the electrical gain of the MPPC, a $430$ nm-wavelength LED333The
wavelength is selected since it is close to the photon detection efficiency
peak of the MPPC is used as the light source. A reverse-biased voltage is
supplied to the MPPC via a low-noise regulated DC power supply. The raw MPPC
signal then is amplified by a NIM RPN-092 PM amplifier 444Access online at
https://www.h-repic.co.jp/pdf/NIM_RPN_090-091.pdf. The amplified signal is
digitized by the DRS4 Evaluation Board 555Access online at
https://www.psi.ch/sites/default/files/2020-02/manual_rev51.pdf and sent
through a USB connection to a DAQ computer. The MPPC digitized pulses are
integrated over a predefined timing window, producing a charge histogram, for
example, as shown in the left of Fig.5. The histogram has peaks corresponding
to the pedestal (readout noise), one p.e., two p.e., etc. pulses of the MPPC.
The output charge equivalent to one incident photon is computed from the peak-
to-peak interval between any consecutive peaks, which are determined by
fitting a multi-Gaussian function to the charge histogram. We measure the
electrical gain with two different types of MPPC and at different values of
the over-voltage. The right of Fig.5 shows that the electrical gains of the
two MPPC generations are varied linearly with the applied over-voltage.
Notably, excellent linearity is achieved even where the environmental
conditions are not controlled strictly. Also, we can have an electrical gain
of $\sim 1.5\times 10^{6}$ if operating MPPC with 3 V over-voltage. The
results are corrected for the electronic gain added from the NIM amplifier and
the DRS4.

Figure 5: The left is a typical charge histogram of a S13081-050CS(X1) MPPC
operated at a given over-voltage. The charge histogram is fitted with a multi-
Gaussian function to extract the output charge equivalent to one
photoelectron. The electrical gain, then obtained by using Eq. 1 with
corrections for using NIM amplifier and the DRS4, is calculated at various
values of the over-voltage and resulted on the right plot. Three different
MPPC pieces per MPPC generation is used to check the reliability and stability
of the measurement.

###  II.2 Thermal and optical noises

There are three noise components induced during the MPPC operation: dark
noise, crosstalk, and afterpulses.

#### Dark noise

The so-called dark noise is induced randomly by the thermal excitation in the
crystal lattice of the MPPC, producing signals indistinguishable from those
due to the photoelectric effect. The dark noise rate is proportional to the
photosensitive area of the MPPC, the over-voltage, and the temperature.
Reducing the noise is of central importance in the MPPC development since,
compared to the vacuum-based photomultiplier, MPPC is relatively noisier. For
the first generation of MPPC, as showed in the following measurement, the high
rate of dark noise of $\sim$ 1 MHz is observed. The noise is suppressed
significantly, a factor of $\sim$10, from the third generation of MPPC. In the
practice of the experiment, the dark noise can be mitigated by applying a
trigger threshold that is significantly higher than the noise level. For the
extremely low-light measurement, like dark matter search, the MPPC dark noise
can be technically suppressed by having MPPC operated in the cryogenic
condition.

#### Measuring the dark noise rate of the MPPC

To minimize the effect of the ambient light, the MPPC is placed inside an
aluminum box and further covered with layers of black sheets. The dark noise
rate, also known as the dark count rate, is defined as the rate of
discriminated pulse above a preset voltage threshold corresponding to 0.5 p.e.
pulse. The voltage threshold is determined according to the over-voltage. In
this measurement, the dark count rates of two generations of MPPC, the first
generation SC10362-13-050C and the third generation S13081-050CS, are
measured. The result is shown in the left plot of Fig. 6. We observe a
difference in the dark count rates between the two MPPC generations. The dark
count rate of the third MPPC generation is significantly lower than the first
one at the same over-voltage. Also, the dark count rates of MPPCs in the first
generation increase dramatically while the rates in the third generation rise
gradually as a function of the over-voltage, showing a better performance of
the latter. The result concludes the excellent linearity of the dark noise
rate on the over-voltage. For practical use of MPPC, the trade-off between the
high electrical gain and the low dark noise rate must be considered.

Figure 6: Measured dark count rate (left) and prompt crosstalk probability
(right) as function of the over-voltage. Two MPPC generations and three MPPC
pieces per each are used for comparison and consistency test.

#### Afterpulses and optical crosstalk

In a given pixel, along with the genuine pulse trigged by the primary
avalanche, the charge carriers, trapped in the same pixel or induced to a
neighboring pixel, can avalanche the second pulse called afterpulse or optical
crosstalk, respectively. Both types of noise are characterized by the relative
timing to the genuine pulse. If the afterpulses are released during the
pixel’s recovery, their amplitudes are smaller than the one p.e. pulse. On the
other hand, if afterpulse is avalanched after the pixel’s recovery, their
amplitudes and shapes are indistinguishable from a genuine pulse. Afterpulses
can be observed on the trail of the left waveform shown in Fig. 3. Prompt
crosstalk, which happens simultaneously with the primary pulse, heightens the
output charge since the pixels are connected in parallel. The two p.e. pulse
observed in Fig. 3 is likely due to the crosstalk. In another scenario, the
crosstalk pulse can be delayed and almost indistinguishable from the delayed
afterpulse.

#### Measuring the prompt crosstalk of the MPPC

The prompt crosstalk probability is estimated and measured utilizing the same
setup as dark count rate measurement. Since we expect a prompt crosstalk pulse
superimposes more than one photon, the crosstalk probability is formulated as
the ratio of dark count rate at 0.5 p.e. and 1.5 p.e. thresholds levels

$\text{Prompt~{}crosstalk~{}probability}=\frac{\text{Dark~{}count~{}rate~{}at~{}1.5~{}p.e.~{}threshold~{}level}}{\text{Dark~{}count~{}rate~{}at~{}0.5~{}p.e.~{}threshold~{}level}}.$
(2)

The result illustrated in the right plot of Fig.6 shows that the crosstalk
probability of the two MPPC generations has the same behavior as the dark
count rate. The crosstalk probabilities are about 10-30$\%$ in the first MPPC
generation and less than $5\%$ in the first 3rd one.

Although the environmental conditions are not strictly controlled, our above
measurements of the electric gain, dark noise rate, and the prompt crosstalk
probability are in good agreement with the measurements in Ref. Fuminao Hosomi
(2016), affirming that MPPC can function well in such conditions.

##  III A tabletop cosmic-ray detector with scintillator and MPPC

To demonstrate the MPPPC’s application in particle and nuclear physics, we
construct a simple tabletop cosmic-ray detector with the plastic scintillator,
the wavelength shifting fiber (WLS), and the MPPC. The schematic setup of the
three-channel detector is shown on the top-left of Fig.7. Each channel is
formed by one extruded 61 cm $\times$ 2.5 cm $\times$ 1.15 cm plastic
scintillator666Considering an integrated cosmic-ray flux of
$1~{}\text{cm}^{-2}\text{min}^{-1}$, $~{}2.5$ events per second is estimated
with this setup. A smaller-sized plastic scintillator can be used but need
more time to collect data for having the same statistics with a 2 mm-diameter
hole in the middle to house the Y-11(200)-typed 1.2-diameter WLS fiber, which
is coupled to one MPPC third-generation S13081-050CS(X1). When charged
particles, like muons, travel through the scintillator, they deposit energy
and produce a flash of scintillation light. The WLS fiber collects these
lights and guides them to the MPPC before detected. A coincident signal of the
top and bottom channels is used as a trigger system to recognize whenever
vertical cosmic rays are coming and acquire the middle one to record the
signal. A typical waveform of the cosmic-ray-like signal is shown in the top-
right of Fig.7. We investigate the MPPC response and decay time of the
scintillation light by fitting the waveform to an exponential-modified
Gaussian function.

Figure 7: The top shows our setup for the cosmic-ray measurement with key
components are highlighted. The bottom left is the schematic diagram of the
setup. The bottom right is a waveform of scintillation photons collected by
the MPPC when a cosmic ray passes through and deposits energy on the plastic
scintillator. The waveforms are fitted with the exponential-modified Gaussian
to estimate the MPPC response and scintillation decay time, showed in Fig. 8.

Figure 8: By fitting the exponential-modified Gaussian functions to the the
waveforms, an example showed in Fig. 7, recorded when cosmic-ray muons passing
through, we obtain distributions of the fitted Gaussian width (left) and decay
time (right) to characterize the MPPC response and the scintillation decay
time, respectively.

The fitted Gaussian width characterizes the MPPC response to the cosmic-ray
deposition and the fitted decay time presents how fast the scintillation light
is produced. The result is shown in the plots of Fig.8. The Gaussian width is
estimated at $6.25\pm 0.18$ ns and the mean value of the fitted decay times is
$27.63\pm 0.19$ ns.

The integrated charge of the signal waveform is used to estimate the number of
photoelectrons corresponding to the energy deposit of cosmic ray muons, which
is about 2 MeV/cm on average Groom  _et al._ (2001). The bottom plots of Fig.
9 show the distributions of the photoelectron number captured by the MPPC for
two different widths of scintillator of 1.15 cm and 2.5 cm. We find that a
Landau function does not provide a good fit to the distribution, but the
convoluted Landau and Gaussian function, affirming the fluctuation of the
energy loss by charged particle’s ionization in a thin layer of material.

Figure 9: The schematic setup (left) to measure the light yield in terms of
the photoelectron (right) captured by the MPPC when the cosmic rays pass
through and deposits energy on 1.15 cm-thick (top) and 2.5 cm-thick (bottom)
plastic scintillators.

In Fig.9, the numbers of detected photoelectrons are $12.3\pm 5.1$ and
$31.1\pm 8.8$ corresponding to the 1.15 cm-thick and 2.5 cm-thick
scintillators, respectively. The result affirms that the energy deposit of the
charged particle is proportional to the thickness of the scintillation
material. Also, it shows that the number of photoelectrons detected when a
cosmic ray passes through is well above the noise level. Thus, we conclude
that MPPC with the plastic scintillation is suitable for tracking the charged
particle in a lab room with loosely controlled conditions.

##  IV SUMMARY

In this paper, we have introduced and measured the characteristics of the MPPC
under loosely controlled conditions. We conclude that the MPPC performance is
still stable and reliable enough for practical use even in such situations. We
also measured the scintillation light yield collected by the MPPC when the
cosmic rays pass through and deposit energy on the plastic scintillator. We
observe more than ten photoelectrons per 2 MeV energy deposition, well above
the background level. The result concludes that MPPC could be used along with
the $\sim 1$ cm or thicker plastic scintillator to construct the simple, cost-
effective tabletop cosmic-ray detector for educational and research purposes.

## References

  * Neamen (2011) D. A. Neamen,  _Semiconductor physics and devices: basic principles_ (McGraw-Hill, 2011).
  * Ghassemi  _et al._ (2018) A. Ghassemi _et al._ ,  _A technical guide to silicon photomultipliers (MPPC) - Overview_, Tech. Rep. (Hamamatsu Photonics K.K., 2018).
  * (3) Hamamatsu Photonics K.K., “Multi-Pixel Photon Counters (MPPCs/SiPMs),” https://www.hamamatsu.com/jp/en/product/optical-sensors/mppc/index.html, [Online; accessed: June 2020].
  * Ohashi (2019) Y. Ohashi, “Current status of hamamatsu si detectors for collider experiments,” https://indico.cern.ch/event/818783/contributions/3598485/ (2019).
  * Yamamoto  _et al._ (2019) K. Yamamoto, T. Nagano, R. Yamada, T. Ito, and Y. Ohashi, JPS Conf. Proc. 27, 011001 (2019).
  * Seifert  _et al._ (2009) S. Seifert, H. T. Van Dam, J. Huizenga, R. Vinke, P. Dendooven, H. Lohner, and D. R. Schaart, IEEE Transactions on Nuclear Science 56, 3726 (2009).
  * Yokoyama  _et al._ (2010) M. Yokoyama _et al._ , Nucl. Instrum. Meth. A 622, 567 (2010), arXiv:1007.2712 [physics.ins-det] .
  * Fuminao Hosomi (2016) Fuminao Hosomi, “Characterization of multi-pixel photon counters for a new neutrino detector,” http://hep.phys.s.u-tokyo.ac.jp/wordpress/wp-content/uploads/2016/06/mth2016_hosomi.pdf (2016), [Online; accessed: May 2021].
  * Groom  _et al._ (2001) D. E. Groom, N. V. Mokhov, and S. I. Striganov, Atom. Data Nucl. Data Tabl. 78, 183 (2001).


# Improved Out-of-Scope Intent Classification with Dual Encoding and
Threshold-based Re-Classification

###### Abstract

Detecting out-of-scope user utterances is essential for task-oriented
dialogues and intent classification. Current methodologies face difficulties
with the unpredictable distribution of outliers and often rely on assumptions
about data distributions. We present the Dual Encoder for Threshold-Based Re-
Classification (DETER) to address these challenges. This end-to-end framework
efficiently detects out-of-scope intents without requiring assumptions on data
distributions or additional post-processing steps. The core of DETER utilizes
dual text encoders, the Universal Sentence Encoder (USE) and the Transformer-
based Denoising AutoEncoder (TSDAE), to generate user utterance embeddings,
which are classified through a branched neural architecture. Further, DETER
generates synthetic outliers using self-supervision and incorporates out-of-
scope phrases from open-domain datasets. This approach ensures a comprehensive
training set for out-of-scope detection. Additionally, a threshold-based re-
classification mechanism refines the model’s initial predictions. Evaluations
on the CLINC-150, Stackoverflow, and Banking77 datasets demonstrate DETER’s
efficacy. Our model outperforms previous benchmarks, achieving an increase of
up to 13% and 5% in F1 score for known and unknown intents on CLINC-150 and
Stackoverflow, and 16% for known and 24% for unknown intents on Banking77. The
source code has been released at https://github.com/Hossam-Mohammed-
tech/Intent_Classification_OOS.

Keywords: Out-of-Scope Detection, Intent Classification, Dual Encoder, USE,
TSDAE

Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-
based Re-Classification

Hossam M. Zawbaa1, 2, Wael Rashwan1, 2, 3, Sourav Dutta4, Haytham Assem5
---
1Technological University Dublin, Ireland.
2 Ireland’s Centre for Applied AI (CeADAR), Ireland
3 3S Group, Faculty of Business, TU DUBLIN, Ireland
4Huawei Research Centre, Ireland
5Amazon Alexa AI, United Kingdom
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

The rise of Generative AI models such as ChatGPT underscores the prominence of
conversational frameworks as the primary interactive interface for end-user
assistance. A crucial goal within these systems is to precisely discern the
user’s intent behind their utterances, a fundamental component for formulating
appropriate responses Coucke et al. (2018). However, the challenge lies in
acquiring comprehensive training data that covers the diverse spectrum of
potential user intents.

A competent intent classifier should identify the predefined intents and
recognize out-of-scope (OOS) data. OOS data are inputs unrelated to the
trained classes, representing queries that do not fit into predefined
categories or intents Cavalin et al. (2020). This is essential for truly
understanding a user’s objective.

Intent classifiers face challenges due to the requirement for domain-specific
labelled datasets, particularly in low-data settings standard in commercial
systems Casanueva et al. (2020). Modern Language models like BERT aid in
efficient intent classification by converting text into vector representations
rich in semantic information. However, current strategies for Out-Of-Scope
(OOS) intent detection often lack generalizability and refinement
opportunities Zhan et al. (2021). To address this, Zhan et al. (2021)
introduced synthetic outlier construction using in-domain training data
embeddings, but these methods are static and offer limited potential for
refining model predictions.

In this paper, we introduce Dual Encoder for Threshold-Based Re-Classification
(DETER), a method for efficient Out-Of-Scope (OOS) intent detection. DETER
combines confidence thresholding and synthetic outlier generation to enhance
intent classification performance Zhan et al. (2021); Cer et al. (2018); Wang
et al. (2021); Maas et al. (2013). Our neural network architecture utilizes
dual sentence encoders: the Universal Sentence Encoder (USE) and Transformer-
based Sequential Denoising Auto-Encoder (TSDAE) Cer et al. (2018); Wang et al.
(2021). We fine-tune the TSDAE model unsupervisedly using Clinc-150 data and
incorporate synthetic outlier generation to enhance OOS intent detection Zhan
et al. (2021).

Summarizing our contributions, this research presents the innovative Dual
Encoder for Threshold-based Re-classification (DETER) optimized for out-of-
scope detection, offering a comprehensive solution devoid of stringent data
distribution assumptions. The system adeptly captures an array of synaptic and
semantic features by utilizing dual text encoding mechanisms, namely USE and
TSDAE, thereby refining user utterance representations and amplifying the
efficiency of out-of-scope detection. A distinctive threshold-based re-
classification mechanism further augments prediction reliability, ensuring a
consistent and precise identification of out-of-scope intents. In addition,
DETER employs a nuanced approach to synthetic outlier generation, merging
self-supervision with open-domain datasets to cultivate a diversified training
set. Empirical assessments on CLINC-150, Stackoverflow, and Banking77 datasets
underscore its superior performance, with notable improvements in F1 scores
relative to established benchmarks. Furthermore, DETER’s streamlined
architecture, with only $1.5$ million trainable parameters, contrasts markedly
with models like Zhan et al. (2021), boasting $125$ million parameters,
showcasing enhanced computational efficiency and scalability without
compromising performance.

## 2\. Related Literature

The choice of embeddings significantly impacts intent classification, with
sentence-level embeddings gaining precedence over token-level ones in Natural
Language Understanding (NLU). However, many specialized NLU models struggle
outside core domains. Graph-based representations, which translate nodes to
vector embeddings, hold promise for enhancing intent detection, particularly
in Out-Of-Scope (OOS) classification. The potential of synthetic datasets,
both artificial and real-world derived, is being explored to improve model
adaptability across diverse data distributions. Techniques such as temperature
scaling and strategic perturbations aid in distinguishing known from unknown
samples. While Transformer models like BERT and RoBERTa have revolutionized
NLU with their OOS resilience, optimization remains crucial. Additionally,
emerging pre-trained dual encoders like USE and ConveRT further augment intent
detection, with evidence suggesting their superiority over refinements such as
BERT-Large.

The selection of embeddings holds critical importance in intent
classification. TEXTOIR, a recent platform introduced for intent detection,
provides tools for open intent detection and discovery tasks, offering a
practical toolbox with expandable interfaces Zhang et al. (2021a). For this
study, we utilized open intent detection to distinguish n-class known intents
and a one-class open intent, adhering to the methodologies and results
published by TEXTOIR, using identical seeds to match intents in evaluating our
DETER framework. OpenMax employs softmax loss and fits a Weibull distribution
to classifier logits Bendale and Boult (2016), while MSP predicts known
classes based on maximum softmax probabilities, excluding those with
probabilities below 0.5 Hendrycks and Gimpel (2018). LOF utilizes density to
pinpoint low-density outliers Breunig et al. (2000), while DOC establishes
multiple probability thresholds for known classes through Gaussian fitting Shu
et al. (2017). DeepUnk integrates margin loss with LOF for adept
identification of unknown classes Lin and Xu (2019), and SEG utilizes a large
margin Gaussian mixture loss for feature representation, offering a unique
approach within deep learning techniques Yan et al. (2020).

Diverging in methodology, ADB, a variant of DA-ADB, favours softmax loss over
traditional distance-aware concepts Zhang et al. (2023a, 2021b). Expanding on
intent representations, the (K+1)-way method merges features from two in-
domain intents Zhan et al. (2021). MDF stands out on the outlier detection
front by employing a one-class SVM post-Mahalanobis distance feature
evaluation Xu et al. (2021). ARPL innovates by maximizing variance between
known-class samples and reciprocal point representations, setting its
detection threshold at 0.5 Chen et al. (2022). KNNCL captures attention by
utilizing KNN, enhancing the learning of semantic features crucial for
detecting out-of-scope intents Zhou et al. (2022).

Training strategies, in particular, often become the defining factor for a
model’s success. A novel approach that has gained traction involves the
infusion of synthetic outliers and the integration of OOS sentences from
varied datasets during training Zhan et al. (2021). Yet, specific strategies,
like threshold introduction, may enhance OOS detection over BiLSTM baseline at
the expense of in-domain performance Hasani et al. (2022). Among these
challenges, deep metric learning emerges as a novel approach in the intent
detection space, fostering enhanced intent representation through techniques
like triplet networks Zhang et al. (2023b). Combined with advanced techniques
like triplet loss and hard samples, these networks provide new precedents in
discriminative intent representation.

## 3\. Dual Encoder for Threshold-based Re-Classification (DETER)

This section describes the proposed DETER framework and discusses its internal
workings.

Given K predefined intent classes $(S_{known})$ in a dialogue system, an
intent detection model aims to predict an utterance’s class, which may be one
of the known intents or an out-of-scope intent. During the training phase, a
set of N-labeled utterances $(D^{tr}_{n})$ is provided for training as shown
below in Equations 1 and 2. This depicts a $K+1$ way classification problem at
the test phase. Previous approaches commonly train a K-way classifier for the
known intents and then attempt to find the outliers.

$S_{known}=\\{C_{i}\\}^{K}_{i=1}$ (1)
$D^{\text{tr}}_{n}=\\{(u_{i},C_{i})\\}^{N}_{i=1}\mid C_{i}\in S_{known}$ (2)

where $C_{i}$ is the $i^{th}$ intent class in the in-domain dataset, $K$ is
the total number of predefined intent classes, and $u_{i}$ is the utterance.

On the other hand, as illustrated in Figure 1, DETER formalizes a (K + 1)-way
classification task in the training phase by constructing out-of-scope
examples through self-supervision and open-domain data without making data
distribution assumptions. The dual encoder-based trained classifier can be
easily used for inference without adaptation or post-processing. In the
following subsections, we describe our proposed approach’s details, including
the dual encoders – Universal Sentence Encoder (USE) and Transformer-based
Denoising AutoEncoder (TSDAE) – representation learning, construction of
pseudo outliers, and training.

Figure 1: Overview of the proposed Dual Encoder for Threshold-based Re-
Classification (DETER).

### 3.1. Universal Sentence Encoder (USE)

USE is a text embedding framework designed to transform text into high-
dimensional vectors. Given a user utterance $u$, the USE embedding function,
denoted by $E_{\text{USE}}$, maps $u$ to a 512-dimensional vector in
$\mathbb{R}^{512}$, i.e., $E_{\text{USE}}:u\rightarrow\mathbb{R}^{512}$.

These embeddings have proven invaluable for many NLP tasks, including text
classification, semantic similarity measurement, and clustering. The pre-
trained USE is publicly accessible on Tensorflow-hub and is available in two
forms, one trained with a Transformer encoder and the other with a Deep
Averaging Network (DAN). The output is a 512-dimensional vector, while the
input is the English text of varying length Cer et al. (2018).

### 3.2. Transformer-based Denoising AutoEncoder (TSDAE)

TSDAE provides unsupervised training of the transformer architecture by
introducing particular types of noise into the input text and subsequently
reconstructing the vector representations (of the noisy input) to their
original input values. During the training, TSDAE encodes broken sentences
into fixed-sized vectors, and the decoder uses these sentence embeddings to
rebuild the original sentence representations. The encoder’s sentence
embedding must accurately capture the semantics for higher-quality
reconstruction.

We tune the TSDAE, built upon a RoBERTa transformer, in an unsupervised mode
using the label-free CLINC-150 dataset, achieving domain adaptation Wang et
al. (2021). At inference, the encoder $E_{\text{TSDAE}}$ transforms variable-
length user utterance,$u$, into a fixed 768-dimensional vector Representation
in $\mathbb{R}^{768}$, $E_{\text{TSDAE}}:u\rightarrow\mathbb{R}^{768}$.

Through noise-induced unsupervised tuning, TSDAE can effectively capture
intricate sentence semantics, making it suitable for intent detection tasks
and discerning user purposes from variable-length user utterances.

### 3.3. Representation Learning

To capture rich linguistic and contextual feature representation from user
utterances, We employed the above two sentence embedding techniques: TSDAE and
USE. Each user utterance, $u$, is independently processed through both the
encoders, yielding a d-dimensional output vector of the TSDAE and USE, as
shown in Equation 3.

$h(u)=\begin{bmatrix}E_{\text{TSDAE}}(u)\\\ E_{\text{USE}}(u)\end{bmatrix}$
(3)

The dimensionality $d$ of $h(u)$ amounts to 1280, which is attained by
concatenating the embedding outputs from TSDAE (768 dimensions) and USE (512
dimensions). As a result, the user utterance transforms into a vector $h(u)$
residing in $\mathbb{R}^{1280}$.

Post this representation learning, the training set is mapped to $D^{tr}_{n}$
in the feature space as shown in equation 4.

$\displaystyle D^{tr}_{n}=\left\\{(h_{i},C_{i})\mid
h_{i}=\begin{bmatrix}E_{\text{TSDAE}}(u_{i})\\\
E_{\text{USE}}(u_{i})\end{bmatrix},\,(u_{i},c_{i})\in
D_{n}\right\\}^{N}_{i=1}$ (4)

### 3.4. Construction of Outliers

We implement the outlier construction method proposed by Zhan et al. (2021),
which involves generating two training outliers: synthetic outliers via self-
supervision and readily available open-domain outliers.

#### Synthetic Outliers

We introduce hard outliers in the feature domain to enhance the model’s
generalization capability for out-of-scope (unknown) intent classification.
These outliers are hypothesized to have representations analogous to
established classes. Leveraging this hypothesis, we propose a self-supervised
methodology to generate these hard outliers using the training set
$D^{tr}_{n}$ as delineated by Zhan et al. (2021).

Specifically, we construct synthetic outliers within the latent space by
utilizing convex combinations of inlier features derived from disparate intent
classes, as depicted in Equation 5.

$h^{oos}=\theta\times h_{\beta}+(1-\theta)\times h_{\alpha}$ (5)

Where $h_{\alpha}$ and $h_{\beta}$ are dual sentence encoder embeddings of two
in-domain utterances randomly sampled from different intent classes in
$D^{tr}_{n}$, and $h^{oos}$ is the constructed synthetic outlier. $\theta$ is
chosen arbitrarily from a uniform distribution $U(0;1)$. To ensure diversity
in synthetic Out-Of-Scope (OOS) training examples, we randomly sample $\theta$
within a specified range. This approach allows us to incorporate "hard,"
"medium," and "easy" examples for each iteration, thereby generating synthetic
outliers that contribute to model training.

#### Open-Domain Outliers

User input in practical dialogue systems can be arbitrary free-form sentences.
Hence, learning architectures can easily use the available open-domain
outliers to mimic real-world outliers and provide learning signals
representing them during training. We thus model arbitrary outliers using the
SQuaD 2.0 question-answering dataset Rajpurkar et al. (2018) Zhan et al.
(2021). Combining the above outliers for training provides robustness to our
framework for OOS detection. Later, we will study the effect of these outliers
on DETER’s performance.

### 3.5. DETER Training Architecture

The model architecture comprises two branches: one processing the Universal
Sentence Encoder (USE) embedding of the input, while the other handles the
Transformer-based TSDAE embedding. This refines the embeddings, aligning
latent representations for similar intents. Each branch independently
undergoes several dense and dropout layers to diminish the embedding dimension
size. Subsequently, the condensed embeddings from both branches are
concatenated before being forwarded to a classification head. Figure 2
illustrates the model’s architecture. The entire system is optimized using
cross-entropy loss against ground-truth intent labels from training data.

Figure 2: The model’s architecture

The aggregate count of trainable parameters in the proposed models amounts to
1,559,808 (1.5 M), in contrast to 124,645,632 (125 M) as observed in the K+1
method by Zhan et al. (2021).

### 3.6. Re-Classification Threshold

A key feature of our DETER framework involves enhancing model predictions
through potential re-classification to enhance performance. In pursuit of this
objective, we adopt the threshold-based approach proposed by Larson et al.
(2019) for the softmax output of the classification model, as illustrated in
Equation 6.

$p(c|u)=softmax(Wh+b)\in R^{K+1}$ (6)

where $W\in R^{(K+1)\times d}$ and $b\in R^{K+1}$ are the classifier’s model
parameters.

At inference time, we first take the prediction output of the model, i.e., the
class $C_{j}$ with the most significant value of $p(c=C_{j}|u)$. However, if
the prediction confidence score of the model for the intent associated with an
utterance ($I(u)$) is greater than a pre-computed threshold $T$, only then the
model output is considered. Else, DETER tags the intent OOS. This is computed
as in Equation 7.

$I(u)=\left\\{\begin{array}[]{ c l }C_{j}&\quad\textrm{if }p(c=C_{j}|u)\geq
T\\\ OOS&\quad\textrm{otherwise}\end{array}\right.$ (7)

It is noteworthy that the threshold $T\in[0,1]$ is calibrated on the
validation set without utilizing any OOS examples. Also, the original OOS
examples from the training dataset were not employed during the training and
validation phases of the DETER framework. Instead, unselected intents and
synthetic outliers were utilized as OOS examples. Furthermore, the threshold
was fine-tuned using the provided validation set.

## 4\. Experimental Setup

We now present DETER’s experimental findings for intent classification. The
main objective is to assign the proper intent labels to utterances in the test
set consisting of in-domain (known) and out-of-scope (unknown) intents. We
separately report the approaches’ macro F1-score performance for the known and
unknown intents. We compare the performance of the proposed DETER framework
with the state-of-the-art methodologies in the TEXTOIR framework Zhang et al.
(2021a).

### 4.1. Dataset

We evaluated the competing approaches on three real-world datasets, CLINC-150,
Banking77, and Stackoverflow, as shown in the table 1.

CLINC-150 consists of 150 intent classes across ten domains designed for out-
of-scope intent detection. The 1,200 OOS sample texts are split as 100 train,
100 validation, and 1,000 test Larson et al. (2019). Banking77 is banking-
specific data with 77 intents from 13,083 customer queries Casanueva et al.
(2020). Stackoverflow features 20 classes, each with 1,000 examples Xu et al.
(2015).

In this study, we consolidate all 1,200 OOS texts from CLINC-150 into the test
set, as our framework does not utilize labelled OOS examples during training
Larson et al. (2019). Experiments conducted with 25%, 50%, and 75% of dataset
intents were repeated five times each to ascertain model robustness and
maintain consistent intents for each run.

### 4.2. Model Hyper-parameters

In our experimental setup, we set the batch size, number of epochs, and
patience for stopping criteria on validation accuracy as 200, 1000, and 100,
respectively. We maintain consistency with the epoch count and patience
interval as the (K+1)-way method Zhan et al. (2021) and replicate the
experimental setup of the TEXTOIR framework Zhang et al. (2021a).
Specifically, we conduct ten experiments for each dataset, randomly selecting
25%, 50%, and 75% of intent classes from the training set as known classes for
training, with the remainder reserved as unknown classes for testing. We
utilize 1200 Out-Of-Scope (OOS) examples from the CLINC-150 dataset as OOS
test samples for all datasets.

Dataset | Intent | Training | Validation | Testing
---|---|---|---|---
Total | 25% | Total | 25% | Total | 25%
Clinc-150 | Known | 15,000 | 3,800 | 3,000 | 760 | 4,500 | 1,140
Unknown | 0 | 11,200 | 0 | 221 | 1,200 | 3,360 + 1,200
Banking77 | Known | 9,003 | 2,119 | 1,000 | 234 | 3,080 | 760
Unknown | 0 | 6,884 | 0 | 221 | 1,200 | 2,320 + 1,200
Stackoverflow | Known | 12,000 | 3,000 | 2,000 | 500 | 6,000 | 1,500
Unknown | 0 | 9,000 | 0 | 221 | 1,200 | 4,500 + 1,200
Table 1: Total number of examples and the computed examples of 25% intent
ratios.

This range is chosen to analyze the impact of varying amounts of synthetic and
open-domain outliers, with optimal results observed around 500 open-domain and
500 synthetic outliers. The (K+1)-way method employs the entire SQuaD 2.0
dataset, comprising nearly 100,000 data points, to generate open-domain
outliers Zhan et al. (2021), establishing the precedent for incorporating
large numbers of outliers. While the SQuaD dataset is open-domain, the
CLINC-150, Stackoverflow, and Banking77 datasets are specific closed-domain
datasets, minimizing the likelihood of overlap. Additionally, the SQuaD
dataset has been previously used as a proxy OOS example for intent
classification Zhan et al. (2021).

For the model optimization, we use $AdamW$ Kingma and Ba (2015) and
categorical cross entropy as a loss function for the multi-class
classification model.

We studied the impact of different outliers during model training using varied
open-domain and synthetic outliers. Notably, on the validation set, we
observed that excessive synthetic outliers adversely affect the open domain
performance. Hence, the number of open-domain outliers varied from [50, 4000]
and synthetic outliers from [50, 16000]. Further, for re-classification of the
model predictions, the threshold $T$, based on the validation set, was set to
$0.7$.

## 5\. Evaluation Results

We extensively compare our proposed DETER framework with existing OOS intent
detection techniques. We consider $3$ different case studies as:

### Evaluating DETER Against State-of-the-Art

We compare our DETER framework with TEXTOIR state-of-the-art open intent
detection techniques, namely (i) OpenMax Bendale and Boult (2016), (ii) DOC
Shu et al. (2017), (iii)ARPL Chen et al. (2022), (iv) DeepUnk Lin and Xu
(2019), (v) SEG Yan et al. (2020), (vi) (K+1)-way Zhan et al. (2021), (vii)
ADB Zhang et al. (2021b), (viii) MSP Hendrycks and Gimpel (2018), (ix) LOF
Breunig et al. (2000), (x) KNNCL Zhou et al. (2022), (xii) MDF Xu et al.
(2021), and (xiii) DA-ADB Zhang et al. (2023a).

For a fair comparison, each technique uses the same BERT model as backbone
Zhang et al. (2021a). Our study uses the TEXTOIR platform to standardize
intent selection for training and testing Zhang et al. (2021a). We used the
same seeds as TEXTOIR for consistent results on benchmark datasets to ensure
identical intents in DETER.

#### Results:

The results from our proposed approach, evaluated at various known intent
ratios (25%, 50%, and 75%) of the total CLINC-150 known intents, are presented
in Table 2. Across all datasets and known intent ratios, the DETER framework
consistently outperforms other methods in known and unknown intent detection.
Notably, in the Clinc-150 and Banking-77 datasets, DETER’s scores for unknown
intents are particularly commendable. Some techniques like KNNCL and LOF face
significant challenges with unknown intents in the Stackoverflow dataset.
Notably, many current methods find it challenging to classify known or unknown
samples with different training intents accurately. For instance, OSS
detection is more efficient in 25% and 50% scenarios, while in-domain
classification is superior for 75%. However, we outperform others in detecting
in-domain and OOS in all scenarios.

DETER consistently achieves an impressive F1 score exceeding 90% for in-domain
and out-of-scope (OOS) classification, even with varying training data,
demonstrating its efficacy in data-scarce scenarios and adaptability to
unfamiliar user inputs. The robust and superior performance of DETER
underscores its capability. We observed that augmenting synthetic data volume
enhances model performance, particularly under limited training data
conditions (e.g., 25% scenario), attributed to improved delineation of OOS
class boundaries with restricted in-domain datasets. Conversely, a balanced
dataset proves more beneficial with extensive training data (e.g., 75% of
intents).

Dataset | Model | Intent Ratio
---|---|---
25% | 50% | 75%
Known | Unknown | Known | Unknown | Known | Unknown
Clinc-150 | (K+1)-way | 74.02 | 90.27 | 81.52 | 84.25 | 86.72 | 79.59
ADB | 77.85 | 92.36 | 85.12 | 88.6 | 88.97 | 84.85
ARPL | 73.01 | 89.63 | 80.87 | 81.81 | 86.1 | 74.67
DA-ADB | 79.57 | 93.2 | 85.58 | 90.1 | 88.43 | 86
DOC | 75.46 | 90.78 | 83.84 | 87.45 | 87.91 | 83.87
DeepUnk | 76.95 | 91.61 | 83.3 | 87.48 | 86.57 | 82.67
KNNCL | 78.85 | 93.56 | 83.25 | 87.85 | 86.14 | 82.05
LOF | 77.77 | 91.96 | 83.81 | 87.57 | 87.24 | 82.81
MDF | 49.43 | 84.89 | 61.6 | 62.31 | 72.21 | 51.33
MSP | 51.02 | 59.26 | 72.82 | 63.71 | 83.65 | 63.86
OpenMax | 62.65 | 77.51 | 79.83 | 82.15 | 71.14 | 75.18
SEG | 46.67 | 59.22 | 62.57 | 61.34 | 42.72 | 40.74
Our (DETER) | 92.19 | 98.42 | 92.02 | 96.26 | 92.15 | 92.51
Banking-77 | (K+1)-way | 67.7 | 82.66 | 77.97 | 72.58 | 85.14 | 59.89
ADB | 70.92 | 85.05 | 81.39 | 79.43 | 86.44 | 67.34
ARPL | 62.99 | 83.39 | 77.93 | 71.79 | 85.58 | 61.26
DA-ADB | 73.05 | 86.57 | 82.54 | 81.93 | 85.93 | 69.37
DOC | 65.16 | 76.64 | 78.38 | 72.66 | 84.14 | 63.51
DeepUnk | 64.97 | 76.98 | 75.61 | 67.8 | 81.65 | 50.57
KNNCL | 65.54 | 79.34 | 75.16 | 67.21 | 81.76 | 51.42
LOF | 62.89 | 72.64 | 76.51 | 66.81 | 84.15 | 54.19
MDF | 44.8 | 85.7 | 64.27 | 57.72 | 75.47 | 33.43
MSP | 50.47 | 39.42 | 73.2 | 46.29 | 84.99 | 46.05
OpenMax | 53.42 | 48.52 | 75.16 | 55.03 | 85.5 | 53.02
SEG | 51.48 | 51.58 | 63.85 | 43.03 | 70.1 | 37.22
Our (DETER) | 87.45 | 97.86 | 88.30 | 95.44 | 87.90 | 93.25
Stackoverflow | (K+1)-way | 50.54 | 52.23 | 70.53 | 51.69 | 81.2 | 45.22
ADB | 77.62 | 90.96 | 85.32 | 87.7 | 86.91 | 74.1
ARPL | 60.55 | 72.95 | 78.26 | 73.97 | 85.24 | 62.99
DA-ADB | 80.87 | 92.65 | 86.71 | 88.86 | 87.66 | 74.55
DOC | 56.3 | 62.5 | 77.37 | 71.18 | 85.64 | 65.32
DeepUnk | 47.39 | 36.87 | 67.67 | 35.8 | 80.51 | 34.38
KNNCL | 41.79 | 15.26 | 61.5 | 8.5 | 76.16 | 7.19
LOF | 40.92 | 7.14 | 61.71 | 5.18 | 76.31 | 5.22
MDF | 48.13 | 83.03 | 62.6 | 50.19 | 73.96 | 28.52
MSP | 42.66 | 11.66 | 66.28 | 26.94 | 81.42 | 37.86
OpenMax | 47.51 | 34.52 | 69.88 | 46.11 | 82.98 | 49.69
SEG | 40.44 | 4.19 | 60.14 | 4.72 | 74.24 | 6
Our (DETER) | 88.16 | 97.35 | 88.82 | 94.71 | 88.16 | 90.35
Table 2: Performance comparison (macro F1-score) of DETER framework for in-domain (known) and OOS (unknown) samples with varying amounts of known training intent ratios (25%, 50%, and 75%) for CLINC-150, Banking-77, and Stackoverflow datasets. Embedding | Intent Ratio
---|---
25% | 50% | 75%
Known | Unknown | Known | Unknown | Known | Unknown
TSDAE (Clinc-150) & USE | 87.45 | 97.86 | 88.30 | 95.44 | 87.90 | 93.25
TSDAE(Roberta) & USE | 65.61 | 89.84 | 70.77 | 78.76 | 84.47 | 83.83
TSDAE (Askubuntu) & USE | 62.43 | 88.52 | 76.23 | 84.61 | 81.77 | 82.09
TSDAE (Scidocs) & USE | 56.32 | 88.39 | 74.77 | 84.53 | 82.17 | 81.45
USE_only | 52.74 | 90.00 | 76.55 | 85.65 | 81.09 | 77.98
Table 3: Performance comparison (macro F1-score) of DETER framework for in-
domain (known) and OOS (unknown) samples with varying amounts of known
training intent ratios (25%, 50%, and 75%) for Banking-77 dataset using
different embedding settings.

### Assessing DETER with Various Embeddings

Our investigation revealed that both the standalone USE model and unsupervised
training of TSDAE using diverse datasets (Clinc-150, Askubuntu, Scidocs, and
original Roberta) exhibited notably inferior performance compared to the
proposed joint architecture of DETER.

Ablation experiments conducted on several dual encoder configurations
(including USE only, TSDAE only without pre-training using Roberta, TSDAE with
unsupervised learning on Clinc-150 combined with USE (DETER), TSDAE with
unsupervised learning on Askubuntu combined with USE, and TSDAE with
unsupervised learning on Scidocs combined with USE) using the Banking77
dataset are detailed in Table 3. From the results presented in Table 3, it is
evident that the DETER framework (TSDAE with unsupervised learning on
Clinc-150 combined with USE) consistently outperforms other configurations in
both known and unknown intent detection tasks. Across all scenarios, DETER
achieves an impressive F1 score exceeding 93% for unknown intents,
representing up to a 10% improvement over the nearest embeddings. Similarly,
for known intents, DETER achieves an F1 score surpassing 87%, exhibiting up to
a 22% improvement over the nearest embeddings.

### Model v/s Threshold Performance

Figure 3 illustrates a comparative analysis of the F1 performance of the model
only (i.e., DETER without the re-classification threshold) and the complete
DETER architecture on the CLINC-150 dataset for classifying known and unknown
intents. Across all intent proportions, the model with threshold consistently
outpaces the standalone version. This emphasizes the positive impact of the
DETER framework on model performance. Notably, the model’s aptitude in
classifying unknown intents is commendable. At the 25% intent proportion, it
reaches a striking 98.46% F1, demonstrating its proficiency in identifying
out-of-scope utterances. Compared to using the model only, the DETER framework
displays improved performance and presents lower standard deviations,
indicative of model robustness. Although the performance gap narrows with
increasing data, the model with the threshold consistently holds an edge. For
known intents, the performance remains relatively stable across varying
proportions, hovering around 91-92%, indicating model consistency in
recognizing the intents it is trained on, irrespective of data volume. Similar
behaviour was demonstrated across the other two datasets, Banking77 and
Stackoverflow, underscoring the robustness and versatility of DETER.

Figure 3: Performance comparison of the ”Model only” versus the ”Model with
threshold (DETER)” on CLINC-150 dataset for both (a) known and (b) unknown
intents across varying intent ratios . The error bars display the standard
deviation across ten runs.

### Class Weights

To tackle the class imbalance issue (between in-domain and OOS training data),
we evaluated DETER with and without class weights in this case study. We
observed no significant performance difference in the two settings. For
example, in the 25% proportion of CLINC-150, the best F1 results obtained with
class weights are 82.07% for the known and 94.32% for the unknown.
Furthermore, without class weights, the performances are 82.10% for known and
94.20% for unknown classes.

## 6\. Discussion

Our research presents DETER, a framework for robust out-of-scope user
utterance detection. DETER offers a comprehensive approach by combining dual
text encoders (USE and TSDAE), a branched deep learning architecture,
synthetic and open-domain outliers, and a threshold-based re-classification
mechanism. We focus on formulating an efficient intent classification
framework, leveraging the combined strength of USE and TSDAE sentence
representations integrated with dense layers.

### DETER’s Performance

DETER’s substantial improvements with limited labelled training sets (25% and
50% cases in Table 2) indicate its enormous potential in real-life
applications. This is particularly significant in an ever-changing landscape
of user interactions where adaptive systems are prominent.

Table 2 shows that the DETER framework performs best on three benchmark
datasets (CLINC-150, Banking77, and Stackoverflow) and outperforms the leading
models. In the CLINC-150 dataset, DETER outperforms the state-of-the-art
models by 13%, 6%, and 3% for different known percentages (25%, 50%, and 75%)
and 5%, 6%, and 6% for different unknown percentages (25%, 50%, and 75%). In
the Banking77 dataset, DETER outperforms the state-of-the-art models by 16%,
7%, 2% for different known percentages and 11%, 13%, and 24% for different
unknown percentages. Similarly, in the Stackoverflow dataset, DETER
outperforms the state-of-the-art models by 7%, 2%, 0.5% for different known
percentages and 6%, 5%, and 15% for different unknown percentages.

This performance improvement of DETER is attributed to its unique combination
of Universal Sentence Encoder (USE) and Transformer-based Sequential Denoising
Auto-Encoder (TSDAE), which offers nuanced sentence embeddings that adeptly
bridge universal linguistic traits with domain-specific nuances. Its branched
architecture bolsters feature extraction, clearly differentiating intent and
OOS utterances. The framework also includes complementary components like
synthetic outliers and threshold-based re-classification mechanisms, further
enhancing its efficacy.

### Synthetic and Open-Domain Outliers

Our findings show that synthetic outliers with higher values have a more
noticeable impact than open-domain outliers, which extend beyond mere
observation. This insight opens up new pathways for research to leverage
synthetic data manipulation to enhance model robustness and performance.

### Efficiency, Scalability, and Applicability

One of the most salient aspects of this model is its remarkable computational
efficiency. The DETER model boasts remarkable computational efficiency with
only $1.5M$ trainable parameters, starkly contrasting the $125M$ in the model
by Zhan et al. (2021). This streamlined design, which does not sacrifice
performance, enhances scalability and flexibility for deployment across
various platforms, even those with limited resources.

### Performance and Robustness

Despite its simplicity of architecture, the model maintained robust
performance, substantiating the efficacy of our approach. This performance
parity with more complex models underlines our approach’s potential to
contribute significantly to ongoing research in deep learning and NLP.

### Limitations and Future Work

Our assessment of the DETER framework encompassed $3$ datasets, providing a
broader evaluation spectrum. However, even with this multi-dataset approach,
inherent data limitations persist. This emphasize the necessity of further
exploration across even more diversified datasets, including multilingual
datasets or languages other than English, to fully evaluate the framework’s
potential. Among future research avenues, few-shot learning is an intriguing
domain poised to augment DETER’s adaptability and efficiency.

## 7\. Conclusions

This work introduces the Dual Encoder for Threshold-based Re-Classification
(DETER), designed to detect out-of-scope user utterances in dialogue systems
and by employing dual text encoders (USE and TSDAE), branched deep learning
neural network architecture, synthetic and open-domain outliers, and a
threshold-based re-classification mechanism, the DETER framework was
rigorously evaluated using CLINC-150, Banking77, and Stackoverflow datasets.
Our findings indicate a promising performance and superiority over the
baseline. Through experimentation with synthetic outliers and open-domain
data, we found that synthetic outliers exerted a more pronounced influence on
the model’s efficacy. This insight underscores the significance of adept
outlier handling to augment the model’s capability. However, introducing class
weights had a negligible impact on the results.

While focusing on three benchmark datasets (CLINC-150, Banking77, and
Stackoverflow) introduces certain constraints, the overarching conclusions
underscore a promising direction for future research. The emphasis lies on
optimizing efficiency, scalability, and applicability without sacrificing
performance. DETER’s evaluation of three benchmark datasets reveals its
superiority, outperforming the state-of-the-art models.

This insight suggests a promising avenue for future research. Given that the
TSDAE framework utilizes either Roberta or BERT models as its backbone, the
possibility of training a multilingual TSDAE using multilingual data arises.
This approach, mirroring the original TSDAE framework by Wang et al. (2021),
could lead to the development of a multilingual DETER framework capable of
handling diverse language datasets.

## 8\. Acknowledgments

This project has received funding from Enterprise Ireland and the European
Union’s Horizon 2020 Research and Innovation Programme under the Marie
Skłodowska-Curie grant agreement No 847402.

## 9\. Bibliographical References

##

  * Agarwal et al. (2021) Vibhav Agarwal, Sudeep Deepak Shivnikar, Sourav Ghosh, Himanshu Arora, and Yashwant Saini. 2021.  Lidsnet: A lightweight on-device intent detection model using deep siamese network.  In _20th IEEE International Conference on Machine Learning and Applications (ICMLA)_ , pages 1112–1117.
  * Bendale and Boult (2016) Abhijit Bendale and Terrance Boult. 2016.  Towards open set deep networks.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.
  * Breunig et al. (2000) Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. 2000.  Lof: Identifying density-based local outliers.  In _Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data_ , page 93–104, New York, NY, USA. Association for Computing Machinery.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45. Association for Computational Linguistics.
  * Cavalin et al. (2020) Paulo Cavalin, Victor Henrique Alves Ribeiro, Ana Appel, and Claudio Pinhanez. 2020.  Improving out-of-scope detection in intent classification by using embeddings of the word graph space of the classes.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3952–3961. Association for Computational Linguistics.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder.  In _arXiv preprint arXiv:1803.11175_.
  * Chen et al. (2022) Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. 2022.  Adversarial reciprocal points learning for open set recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(11):8065–8081.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018.  Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces.  In _arXiv preprint arXiv:1805.10190_ , pages 12–16.
  * Hasani et al. (2022) Muhammad Fikri Hasani, Ford Lumban Gaol, Benfano Soewito, and Harco Leslie Hendric Spits Warnars. 2022.  Deep learning and threshold probability for out of scope intent detection in task oriented chatbot.  In _3rd International Conference on Artificial Intelligence and Data Sciences (AiDAS)_ , pages 322–327.
  * Hendrycks and Gimpel (2018) Dan Hendrycks and Kevin Gimpel. 2018.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _arXiv, 1610.02136_.
  * Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020.  Pretrained transformers improve out-of-distribution robustness.  In _In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2744–2751. Association for Computational Linguistics.
  * Kartsaklis et al. (2018) Dimitri Kartsaklis, Mohammad Taher Pilehvar, and Nigel Collier. 2018.  Mapping text to knowledge graph entities using multi-sense LSTMs.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1959–1970. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations (ICLR)_ , San Diego, CA, USA.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316. Association for Computational Linguistics.
  * Liang et al. (2020) Shiyu Liang, Yixuan Li, and R. Srikant. 2020.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In _arXiv preprint arXiv: 1706.02690_.
  * Lin and Xu (2019) Ting-En Lin and Hua Xu. 2019.  Deep unknown intent detection with margin loss.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5496, Florence, Italy. Association for Computational Linguistics.
  * Maas et al. (2013) Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013.  Rectifier nonlinearities improve neural network acoustic models.  In _Proceedings of the 30th International Conference on Machine Learning_ , Atlanta, Georgia, USA.
  * Orhan (2019) A. Emin Orhan. 2019.  Robustness properties of facebook’s resnext wsl models.  In _arXiv preprint arXiv: 1907.07640_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992. Association for Computational Linguistics.
  * Shu et al. (2017) Lei Shu, Hu Xu, and Bing Liu. 2017.  DOC: Deep open classification of text documents.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2911–2916, Copenhagen, Denmark. Association for Computational Linguistics.
  * Wang et al. (2021) Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021.  Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 671–688. Association for Computational Linguistics.
  * Xu et al. (2015) Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015.  Short text clustering via convolutional neural networks.  In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_ , pages 62–69, Denver, Colorado. Association for Computational Linguistics.
  * Xu et al. (2021) Keyang Xu, Tongzheng Ren, Shikun Zhang, Yihao Feng, and Caiming Xiong. 2021.  Unsupervised out-of-domain detection via pre-trained transformers.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1052–1061. Association for Computational Linguistics.
  * Yan et al. (2020) Guangfeng Yan, Lu Fan, Qimai Li, Han Liu, Xiaotong Zhang, Xiao-Ming Wu, and Albert Y.S. Lam. 2020.  Unknown intent detection using Gaussian mixture model with an application to zero-shot intent classification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1050–1060, Online. Association for Computational Linguistics.
  * Zhan et al. (2021) Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao-Ming Wu, and Albert Y.S. Lam. 2021.  Out-of-scope intent detection with self-supervision and discriminative training.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3521–3532. Association for Computational Linguistics.
  * Zhang et al. (2021a) Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. 2021a.  TEXTOIR: An integrated and visualized platform for text open intent recognition.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 167–174, Atlanta, Georgia, USA.
  * Zhang et al. (2021b) Hanlei Zhang, Hua Xu, and Ting-En Lin. 2021b.  Deep open intent classification with adaptive decision boundary.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14374–14382.
  * Zhang et al. (2023a) Hanlei Zhang, Hua Xu, Shaojie Zhao, and Qianrui Zhou. 2023a.  Learning discriminative representations and decision boundaries for open intent detection.  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , volume 31, pages 1611–1623.
  * Zhang et al. (2023b) Youwen Zhang, Xudong Wang, Linlin Wang, Ke Yan, and Huan Chen. 2023b.  Out-of-scope intent detection with supervised deep metric learning.  In _International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8.
  * Zhou et al. (2022) Yunhua Zhou, Peiju Liu, and Xipeng Qiu. 2022.  KNN-contrastive learning for out-of-domain intent classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5129–5141. Association for Computational Linguistics.