# TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language
Navigation

###### Abstract

Zero-shot navigation is a critical challenge in Vision-Language Navigation
(VLN) tasks, where the ability to adapt to unfamiliar instructions and to act
in unknown environments is essential. Existing supervised learning-based
models, trained using annotated data through reinforcement learning, exhibit
limitations in generalization capabilities. Large Language Models (LLMs), with
their extensive knowledge and emergent reasoning abilities, present a
potential pathway for achieving zero-shot navigation. This paper presents a
VLN agent based on LLMs, exploring approaches to the zero-shot navigation
problem. To compensate for the shortcomings of LLMs in environmental
perception, we propose the Thinking, Interacting, and Action (TINA) framework.
TINA enables the agent to scrutinize perceptual information and autonomously
query key clues within the environment through an introduced question-
answering module, thereby aligning instructions with specific perceptual data.
The navigation agent’s perceptual abilities are enhanced through the TINA
framework, while the explicit thought and query processes also improve the
navigational procedure’s explainability and transparency. We evaluate the
performance of our method on the Room-to-Room dataset. The experiment results
indicate that our approach improves the navigation performance of LLM-based
agents. Our approach also outperformed some supervised learning-based methods,
highlighting its efficacy in zero-shot navigation.

Index Terms—  Zero-shot, Navigation, Vision and language, Agent, Large
language model

##  1 Introduction

The advancement of computer vision and natural language processing has
facilitated research in vision-language fusion, such as Visual Question
Answering[1] (VQA), Image Caption[2], and Vision-Language Navigation[3] (VLN).
In VLN tasks, agents must navigate through diverse environments based on
natural language instructions, necessitating multifaceted expertise in
linguistic semantics, visual perception, and dynamic decision-making (Figure
1). While supervised deep learning has propelled the development of VLN
models, existing models exhibit limitations in generalization and decision-
making transparency[4, 5, 6, 7, 8]. They still lack the zero-shot ability[9]
to interpret unfamiliar instructions and navigate unseen environments, which
limits the development of these models for broader applications.

Fig. 1: (a) A VLN Examples. (b) Supervised learning methods train a policy
network based on pre-trained visual encoders and text embeddings. (c) LLM-
based agents utilize an LLMs for reasoning. Our framework introduces
additional modules to enhance the agent’s capabilities. The snowflake symbol
indicates frozen parameters, the flame signifies trainable ones.

Large Language Models (LLMs) have recently received considerable attention due
to their remarkable language generation capabilities and extensive
knowledge[10, 11]. Research indicates that as LLMs scale, they demonstrate
emergent abilities that expand their applicability across various domains,
including reasoning and decision-making tasks[12]. Recent research has
revealed that, despite being trained, current VLN models still encounter
difficulties when dealing with diverse instructions[13]. For the VLN task,
LLMs hold promise for enabling the zero-shot capability[14, 15, 16]. They can
leverage their extensive knowledge and common sense reasoning to interpret and
decompose new instructions and reason according to the environment, thus
enabling navigation agents to adapt to unseen environments without prior
training. However, a challenge arises in developing LLM-based VLN systems due
to LLMs’ limited visual perceptual capabilities[17], primarily trained on
textual data. Some efforts introduced visual perceptual modules, converting
visual information into textual descriptions for analysis by LLMs[18, 19, 20].
While these approaches give LLM-based agents a rudimentary understanding of
visual content, descriptions often remain generalized and lack specificity. In
cases where visual perception needs to align with specific instructions, these
visual descriptions might omit crucial information.

Fig. 2: The schematic diagram of the TINA framework. It primarily consists of
the core LLM-based agent and three peripheral modules. 1) The Visual
Perception module is used to acquire descriptions of the surrounding
environment and distance information related to various objects. 2) The
Question-Answering Interaction module performs targeted clue queries on
Candidates’ visual images based on the agent’s reasoning Thoughts. 3) The
Trajectory Memorizer summarizes observations and actions from this step,
storing new memories in the memory bank.

To enhance the perceptual capabilities of LLMs for specific targets, we
propose the TINA framework — Think, Interaction, and Action — which endows
agents with the ability to scrutinize perceptual outcomes and autonomously
query specific clues. Our framework comprises the core LLM agent and three
additional modules: the Visual Perception (VP) module, the Question-Answer
Interaction (QAI) module, and the Trajectory Memorizer (TM) module. The VP
module generates rough environmental descriptions, prompting the agent to
reason based on these descriptions in response to specific perceptual demands
in instructions. The framework enables the agent to review the outputs of the
VP and make targeted queries through the QAI module to supplement missing
perceptual information. TINA extends the agent’s perception through the QAI
module, aligning instructions with the environment. Additionally, our
framework includes a memory bank that stores the agent’s actions in each
round, enhancing its dynamic adaptation capabilities while filtering out
redundant historical information. In the subsequent sections, we will provide
a detailed description of our TINA framework for zero-shot VLN, along with
insights from our experimental results and findings.

##  2 Method

###  2.1 Preliminaries

For a specific environment, there exists a navigation graph $G=(V,E)$,
comprised of a viewpoint set $V$ and a path set $E$. The VLN task requires the
agent to move from an initial viewpoint to a designated target viewpoint based
on a navigation instruction $I$. The agent’s action at each step is to choose
the next viewpoint. The viewpoints visited during navigation are sequentially
constructed into a trajectory $R=(v_{0},v_{1},...,v_{T})$, also representing
the sequence of agent actions. When the agent arrives at a specific viewpoint
$v_{t}$, it can access the panoramic observation $O_{t}$ and a navigable
viewpoint set $C_{t}$. The panoramic observation
$O_{t}=\\{(o_{t,p},h_{t,p}^{r},e_{t,p}^{r})\\}$ encompasses local information
obtained from diverse directions, where $h^{r}$ and $e^{r}$ represent the
heading angle and elevation angle relative to the agent, $o_{t,p}$ denotes the
local view in that direction. Set $C_{t}$ encompasses adjacent viewpoints
$\\{(v_{q},o^{c}_{q},h_{q}^{r},e_{q}^{r})|v_{q}\in\mathcal{N}(v_{t}|G)\\}$,
$o^{c}_{q}$ is the local view in the direction of $v_{q}$. At step $t$, the
agent selects a viewpoint from $C_{t}$ as the action $v_{t}$, and then adjusts
its orientation and position to update its status from $s_{t}$ to
$s_{t+1}=(v_{t+1},h_{t+1},e_{t+1})$, where $h$ and $e$ indicate the
orientation of the agent itself. The agent can be regarded as a policy
function for computing the probability
$P(v_{t}|s_{t},O_{t},C_{t},R_{[:t-1]}I;\Theta)$, wherein $\Theta$ is the
function’s parameters. In supervised learning, the parameters $\Theta$ are
optimized by calculating the loss between the ground truth and predicted
trajectory. This study uses a pre-trained LLM with frozen parameters to
explore the agent’s performance in a zero-shot setting. Figure 1 illustrates
these two approaches’ differences.

###  2.2 Framework Overview

We propose an innovative agent framework for VLN agents, comprising a core LLM
and three auxiliary modules: Visual Perception ($\mathrm{VP}$), Question-
Answering Interaction ($\mathrm{QAI}$), and Trajectory Memorizer
($\mathrm{TM}$). The LLM-based agent is the central component of our
framework, responsible for high-level reasoning and predicting the final
actions.

At each step $t$ in the trajectory, the agent can obtain panoramic visual
images from the environment. The Visual Perception module transforms the
incoming panoramic visual data into initial textual descriptions denoted as
$d_{t}$. The LLM-based agent conducts reasoning based on global instruction
and the visual description $d_{t}$, generating intermediate reasoning results
represented as Thought $h_{t}$. The core of this framework lies in the
Question-Answering Interaction module, which plays a critical role in
assessing the compatibility between potential candidate viewpoints and the
agent’s Thoughts. It queries the local views of the candidate viewpoints
through a question-and-answer interaction, searching for crucial clues aligned
with the reasoning. The query results are integrated into the descriptions of
the candidate viewpoints, supplementing the information overlooked by the VP
module. Subsequently, the agent selects one viewpoint from the enhanced set of
candidate viewpoints as action $v_{t}$.

$\displaystyle d_{t}$ $\displaystyle=\mathrm{VP}(O_{t})$ (1) $\displaystyle
h_{t}$ $\displaystyle=\mathrm{LLM}(d_{t}|I,M_{[:t-1]})$ (2) $\displaystyle
d^{c}_{t}$ $\displaystyle=\mathrm{QAI}(h_{t},C_{t})$ (3) $\displaystyle v_{t}$
$\displaystyle=\mathrm{LLM}(d^{c}_{t}|I,M_{[:t-1]})$ (4)

To ensure the agent understands the task requirements, we provide a detailed
description in the prompt, including task setup, module introductions, and
input-output formats. Following the task description, we also offer a global
instruction for each navigation case. As the agent moves through the
environment, the execution history of its trajectory impacts its decision-
making[21]. Therefore, we establish a textual memory bank $M$ to record the
agent’s historical actions and input it into the agent as a prompt. These
three components together form the preliminary prompt, which is conveyed to
the agent at the beginning of each step before it receives new observations,
as illustrated in Figure 2.

###  2.3 Observation Snapshot

The Visual Perception (VP) module translates visual information from the
environment into textual descriptions that LLMs can understand. To enable the
agent to perceive information from different directions, we ask the agent to
gather visual images from 24 different directions, forming a panoramic
observation (eight directions horizontally, with images captured at intervals
of 45 degrees, and three directions vertically, covering upward / downward 30
degrees, and straight ahead at 0 degrees). Initially, we employ the BLIP-2
image captioning model[22] to generate textual descriptions for each of the 24
directions. We have observed that in most cases, the agent primarily relies on
information from the straight-ahead view for navigation. Therefore, we use
another LLM to consolidate the content of the three descriptions in the
vertical direction, resulting in eight distinct descriptions for navigation in
different directions.

Fig. 3: The figure shows how to obtain an object’s distance.

We use IOU to match the results of DETR object detection[23] and Mask2Former
segmentation[24] to enhance the agent’s perception of distance to surrounding
objects. We obtain the pixels of object $i$ in the RGB image by intersecting
the bounding box $b_{i}$ and the mask $k_{i}$, then map them to the depth map,
as shown in Figure 3. We calculate the average value of the pixels in the
depth map as the distance $l_{i}$ between object $i$ and the agent:

$\displaystyle l_{i}$ $\displaystyle=\frac{1}{|S_{i}|}\Sigma_{(x,y)\in
S_{i}}D_{(x,y)}$ (5) $\displaystyle S_{i}$ $\displaystyle=b_{i}\cap k_{i}$ (6)

where $D_{(x,y)}$ is the value of the coordinate $(x,y)$ in the depth map. We
filter objects within a 3-meter range and incorporate their distance
information into the corresponding directional descriptions.

###  2.4 Interaction for Candidates Investigation

The VP module provides the agent with environmental awareness by analyzing
images from multiple discrete directions. However, this approach poses two
problems for VLN. First, candidates may be located in arbitrary directions
relative to the agent, leading to the segmentation of images in that direction
by the VP module. To address this, we can generate descriptions again by
invoking the VP module specifically for the candidate’s direction. Second, the
VP module provides generic descriptions and does not dynamically generate
relevant information based on the agent’s reasoning state, which results in
incomplete environmental perception by the agent and limits the practical
significance of its reasoning. This is also a challenge faced by existing LLM-
based agents[25].

Therefore, we propose the Question Answering Interaction (QAI) module.
Inspired by ReAct[26], our approach adopts a pattern of interleaved reasoning
and action execution, allowing the agent to dynamically create, maintain, and
adjust high-level action plans presented as Thought. This approach enhances
the model’s interpretability and distinguishes between the model’s internal
knowledge and external environmental information in scenarios involving
interaction with the external environment, thus revealing the agent’s need for
external information during navigation. We input the agent’s generated
thoughts into the QAI module, which analyzes and summarizes visual cues
related to these thoughts, constructing corresponding visual questions.
Answers to these questions are obtained through a Visual Question Answering
model from different candidate image views, as shown in Figure 4. The
question-answer pairs are organized and added as supplementary content to the
descriptions of the corresponding candidates. Ultimately, the agent selects an
appropriate viewpoint from the optimized candidates set as current round
action.

The QAI module makes full use of the agent’s Thoughts, acquiring directional
perceptual information and enhancing the coupling between visual perception
and LLM reasoning.

###  2.5 Trajectory Memorizer

In VLN tasks, agents need to adjust their navigation strategies based on
historical data dynamically. However, directly inputting all historical data
into an LLM is impractical. In the later stages of navigation, the accumulated
data from historical observations, reasoning, and actions can exceed the
length limitations of the LLM. Additionally, a significant amount of redundant
information can reduce the efficiency and accuracy of the agent’s access to
helpful information. To enable efficient access to the agent’s historical
trajectories, we introduce a memory bank $M$. After each round of reasoning,
interaction, and action execution, we use a trajectory memorizer to compress
and summarize the proceedings and store this new memory $m_{t}$ in the memory
bank.

$\displaystyle m_{t}=\mathrm{TM}(d_{t},h_{t},d^{c}_{t},v_{t})$ (7)

Although the memory bank grows as the trajectories accumulate, it is far
smaller than the storage requirements for the entire dataset. It also filters
out irrelevant information from history, retaining only key points.

##  3 Experiment

The TINA framework is implemented based on gpt-4[27] model. We conducted
experiments on the R2R dataset[3], comparing the model’s performance on the
validation unseen split with previous methods. We also randomly sampled a
portion of the entire dataset to create a dev split for analyzing the
framework’s modules. We adopt the following navigation metrics for evaluation:
Trajectory Length (TL): average path length in meters; Navigation Error (NE):
average distance in meters between the final and target location; Success Rate
(SR): the ratio of paths with NE less than 3 meters; Oracle SR (OSR): SR given
oracle stop policy; SR penalized by Path Length (SPL). The Results are shown
in Table 1 and Table 2.

Fig. 4: The key prompts and their structure for Agent navigation. The handwritten text is content generated by LLM. The numbers indicate the sequence in which the modules operate. Table 1: Performance on R2R validation unseen split Model | ZS | TL | NE $\downarrow$ | OSR $\uparrow$ | SR $\uparrow$ | SPL $\uparrow$
---|---|---|---|---|---|---
Seq2Seq[3] | ✗ | 8.39 | 7.81 | 28 | 21 | -
SF[6] | ✗ | - | 6.62 | 45 | 35 | -
Chasing[8] | ✗ | 9.64 | 7.20 | 44 | 35 | 31
EnvDrop[7] | ✗ | 10.70 | 5.22 | - | 52 | 48
LangNav[28] | ✓ | - | 7.0 | 42 | 32 | 28
NavGPT[20] | ✓ | 11.45 | 6.46 | 42 | 34 | 29
TINA | ✓ | 11.37 | 5.93 | 48 | 37 | 33

Table 1 compares TINA with several existing methods, including supervised
learning baselines and the latest zero-shot work. The ”ZS” in the table
indicates whether the method is zero-shot. The experimental results show that
TINA has surpassed some supervised learning methods and outperformed the
latest zero-shot methods, demonstrating its potential in zero-shot navigation.
We also conducted several ablation experiments to investigate the roles of
different components in the TINA framework. We randomly sampled a small subset
of the entire dataset to create a dev split for the ablation experiments, and
the results are presented in Table 2.

First, we investigated the functionality of the QAI module. In this setting
(”$w/o$ QAI”), the agent does not ask questions about the image content and
relies solely on the VP’s glancing descriptions for navigation. It can be
observed that without the QAI module, the agent may lose crucial information,
leading to a decline in navigation performance, which highlights the
importance of expanding the visual perception capabilities of LLM-based
agents.

Table 2: Ablation experiments results Setting | TL | NE $\downarrow$ | OSR $\uparrow$ | SR $\uparrow$ | SPL $\uparrow$
---|---|---|---|---|---
base | 12.03 | 7.25 | 40 | 31 | 27
-w/o QAI | 12.88 | 7.81 | 37 | 29 | 26
-w/o dis | 14.91 | 8.43 | 32 | 25 | 22
-w/o seg | 13.14 | 7.49 | 36 | 29 | 26

To explore the impact of distance perception on agent navigation, we attempted
to turn off the agent’s perception of distance, shown as ”$w/o~{}dis$”. It can
be seen that the absence of distance information results in a significant
performance drop. When the VP stops providing distance information between
objects, the scene depicted by the environment description will have greater
randomness, and the agent also struggles to estimate the distance between
itself and the target point. We further tested the performance gain using
instance segmentation in the VP (see ”$w/o~{}seg$”). In this experiment, we
only obtained depth information through the center points of bounding boxes,
decreasing the agent’s navigation success rate. We attribute this to the
irregular shapes of objects, which can cause the center point of the bounding
box to fall outside the object pixels, resulting in inaccurate distance
calculations. Segmentation allows for a more precise alignment of RGB and
depth pixels.

Through the ablation experiments, we have demonstrated that environmental
perception is the most crucial factor limiting the zero-shot navigation of
LLM-based agents. We propose TINA to enhance the agent’s environmental
perception capabilities, and the experiments indeed confirm the effectiveness
of TINA. Additionally, we have found that LLM-based agents require cognitive
and modeling capabilities in 3D space to adapt to a broader range of tasks.
Despite some enhancements we have made to improve the agent’s accuracy in
obtaining distance information, the transition from 2D perception to 3D
perception remains an important future research direction for LLM-based
agents.

We also conducted a case study on the agent’s and QAI’s interactions, and some
of the results are shown in Figure 5. It can be observed that the QAI module
can generate descriptions closely related to Thoughts, enabling the agent to
make more accurate selections for the next viewpoint, which also enhances the
explainability of the navigation process.

##  4 Conclusion

In this paper, we explore the zero-shot navigation problem based on LLMs and
propose the TINA framework, enabling the agent to scrutinize perceptual
information and autonomously query key clues, thereby enhancing the agent’s
perceptual capabilities. Experimental results validate the effectiveness of
our approach, which surpasses existing zero-shot navigation models and some
supervised learning-based methods without the need for additional training. We
also discuss the roles of each module and present examples of explainability
brought about by the QAI module.

Fig. 5: Some candidate viewpoints selected by the agent based on the Thought,
along with the corresponding QAI output.

## References

  * [1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh,  “Vqa: Visual question answering,”  in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425–2433.
  * [2] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio,  “Show, attend and tell: Neural image caption generation with visual attention,”  in International conference on machine learning. PMLR, 2015, pp. 2048–2057.
  * [3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel,  “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3674–3683.
  * [4] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould,  “Vln bert: A recurrent vision-and-language bert for navigation,”  in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2021, pp. 1643–1653.
  * [5] Bingqian Lin, Yi Zhu, Zicong Chen, Xiwen Liang, Jianzhuang Liu, and Xiaodan Liang,  “Adapt: Vision-language navigation with modality-aligned action prompts,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15396–15406.
  * [6] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell,  “Speaker-follower models for vision-and-language navigation,”  Advances in Neural Information Processing Systems, vol. 31, 2018.
  * [7] Hao Tan, Licheng Yu, and Mohit Bansal,  “Learning to navigate unseen environments: Back translation with environmental dropout,”  arXiv preprint arXiv:1904.04195, 2019.
  * [8] Peter Anderson, Ayush Shrivastava, Devi Parikh, Dhruv Batra, and Stefan Lee,  “Chasing ghosts: Instruction following as bayesian state tracking,”  Advances in neural information processing systems, vol. 32, 2019.
  * [9] Bernardino Romera-Paredes and Philip Torr,  “An embarrassingly simple approach to zero-shot learning,”  in International conference on machine learning. PMLR, 2015, pp. 2152–2161.
  * [10] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.,  “Llama: open and efficient foundation language models, 2023,”  URL https://arxiv. org/abs/2302.13971, 2023.
  * [11] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.,  “Chain-of-thought prompting elicits reasoning in large language models,”  Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022.
  * [12] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.,  “Emergent abilities of large language models,”  arXiv preprint arXiv:2206.07682, 2022.
  * [13] Yue Zhang and Parisa Kordjamshidi,  “Vln-trans: Translator for the vision and language navigation agent,”  arXiv preprint arXiv:2302.09230, 2023.
  * [14] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa,  “Large language models are zero-shot reasoners,”  Advances in neural information processing systems, vol. 35, pp. 22199–22213, 2022.
  * [15] Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng, Thomas H Li, Gaowen Liu, Mingkui Tan, and Chuang Gan,  “$a^{2}$ nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models,”  arXiv preprint arXiv:2308.07997, 2023.
  * [16] Xiwen Liang, Liang Ma, Shanshan Guo, Jianhua Han, Hang Xu, Shikui Ma, and Xiaodan Liang,  “Mo-vln: A multi-task benchmark for open-set zero-shot vision-and-language navigation,”  arXiv preprint arXiv:2306.10322, 2023.
  * [17] William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh,  “Towards language models that can see: Computer vision through the lens of natural language,”  arXiv preprint arXiv:2306.16410, 2023.
  * [18] Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al.,  “Internchat: Solving vision-centric tasks by interacting with chatbots beyond language,”  arXiv preprint arXiv:2305.05662, 2023.
  * [19] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny,  “Minigpt-4: Enhancing vision-language understanding with advanced large language models,”  arXiv preprint arXiv:2304.10592, 2023.
  * [20] Gengze Zhou, Yicong Hong, and Qi Wu,  “Navgpt: Explicit reasoning in vision-and-language navigation with large language models,”  arXiv preprint arXiv:2305.16986, 2023.
  * [21] Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev,  “History aware multimodal transformer for vision-and-language navigation,”  Advances in neural information processing systems, vol. 34, pp. 5834–5847, 2021.
  * [22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi,  “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,”  arXiv preprint arXiv:2301.12597, 2023.
  * [23] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko,  “End-to-end object detection with transformers,”  in European conference on computer vision. Springer, 2020, pp. 213–229.
  * [24] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar,  “Masked-attention mask transformer for universal image segmentation,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 1290–1299.
  * [25] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li,  “Instruct2act: Mapping multi-modality instructions to robotic actions with large language model,”  arXiv preprint arXiv:2305.11176, 2023.
  * [26] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao,  “React: Synergizing reasoning and acting in language models,”  arXiv preprint arXiv:2210.03629, 2022.
  * [27] OpenAI,  “Gpt-4 technical report,” 2023.
  * [28] Bowen Pan, Rameswar Panda, SouYoung Jin, Rogerio Feris, Aude Oliva, Phillip Isola, and Yoon Kim,  “Langnav: Language as a perceptual representation for navigation,”  arXiv preprint arXiv:2310.07889, 2023.


11institutetext: Institute of Computing Science, Poznan University of
Technology, Poznan, Poland 22institutetext: Institute of Bioorganic Chemistry,
Polish Academy of Sciences, Poznan, Poland 33institutetext: MNM Bioscience,
Cambridge, MA
33email<EMAIL_ADDRESS>
#  confidence-planner: Easy-to-Use Prediction Confidence Estimation and Sample
Size Planning

Antoni Klorek 11 0000-0001-5851-4045 Karol Roszak 11 0000-0002-0786-5053
Izabela Szczech 11 0000-0002-9655-4109 Dariusz Brzezinski(✉) 112233
0000-0001-9723-525X

###### Abstract

Machine learning applications, especially in the fields of medicine and social
sciences, are slowly being subjected to increasing scrutiny. Similarly to
sample size planning performed in clinical and social studies, law-makers and
funding agencies may expect statistical uncertainty estimations in machine
learning applications that impact society. In this paper, we present an easy-
to-use python package and web application for estimating prediction confidence
intervals. The package offers eight different procedures to determine and
justify the sample size and confidence of predictions from holdout, bootstrap,
cross-validation, and progressive validation experiments. Since the package
builds directly on established data analysis libraries it seamlessly
integrates into preprocessing and exploratory data analysis steps. Code
related to this paper is available at: https://github.com/dabrze/confidence-
planner

##  1 Introduction

Medical, social, and behavioral sciences are known to be plagued by
undersampling [5]. In the traditional statistical framework, even when the
effect exists, undersampled studies yield either non-significant results or
significant results because of overestimating the size of the effect. Similar
problems can occur in machine learning studies on life-science data, where
classification accuracy is often measured on relatively small samples without
providing any uncertainty estimation. Importantly, statistical testing and
uncertainty estimation of machine learning systems will become more and more
common, as the global community realizes that AI systems need to be controlled
to maintain fairness and comply with government regulations. Examples of such
regulations have recently appeared in the proposed EU AI act [4], UNESCO
recommendation on ethics in AI [8], or the FDA’s Software as Medical Device
guidance [1].

To mitigate issues with undersampled studies, in social sciences authors are
increasingly expected to plan and justify the sample size of their study. Such
sample-size justification procedures can be found in tutorial papers and
online tools [5]. Equivalent confidence interval and sample size estimation
procedures for classification accuracy are scattered throughout scientific
literature [6, 2, 3, 7] and, to the best of our knowledge, are not available
as a python package. The confidence-planner package presented herein aims to
fill this gap.

##  2 The confidence-planner Package

The confidence-planner package provides implementations of estimation
procedures for confidence intervals around classification accuracy. A
confidence interval (CI) is a range of estimates for an unknown parameter. In
our case it is the range of values that we expect the accuracy $\mathit{acc}$
of our model to fall between if we re-run our experiment again. A confidence
interval is computed for a test sample size $n$ and at a designated confidence
level $\gamma$, which is the percentage of times one expects to reproduce an
estimate between the upper and lower bounds of the confidence interval. A
interval with a confidence level of 90% is called a 90% confidence interval
(90% CI).

For a given number of test samples $n$, test accuracy $\mathit{acc}$, and
expected confidence level $\gamma$, confidence-planner offers a set of CI
estimation procedures. The package currently features approximations for
holdout (Langford [6], Clopper-Pearson [3], Wilson [9], Z-test, t-test),
bootstrap [7], cross-validation [2], and progressive validation [2]
experiments. Moreover, for selected methods, for a given confidence level
$\gamma$, confidence-planner can help estimate the number of samples $n$
needed to obtain a CI of a user-specified radius. For example, using the
Z-test approximation, confidence-planner can help estimate that in order to
achieve a 90% CI of $\pm 0.05$ one needs a holdout test of at least 271
examples.

The confidence-planner package is open source and available under a permissive
MIT license. The source code, documentation and an introductory video are
available at https://github.com/dabrze/confidence-planner. The package can
also be installed via PyPI using pip install confidence-planner. The package’s
basic functionality, along with guidance on selecting the appropriate
estimation method, is also available the form of a confidence-planner web
application that can be deployed using the code in the repository.

##  3 Application Example

Next, we present a basic application example featuring the well-known Breast
Cancer Wisconsin dataset to demonstrate how confidence interval estimation can
be included in a data classification script. In this particular example, we
will estimate the 90% confidence interval (CI) of a classifier tested on a
holdout test set using a Z-test approximation [6]. Then, we will estimate the
holdout size that would be required to limit the 90% CI to a radius 0.05
classification accuracy (estimated accuracy $\pm 0.05$). The complete code
required to execute these tasks is the following:

⬇

from sklearn import datasets, svm, metrics

from sklearn.model_selection import train_test_split

import confidence_planner as cp

# example dataset

X, y = datasets.load_breast_cancer(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.3, stratify=y, random_state=23

)

# training the classifier and calculating accuracy

clf = svm.SVC(gamma=0.001)

clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)

acc = metrics.accuracy_score(y_test, y_pred)

# confidence interval and sample size estimation

ci = cp.estimate_confidence_interval(

len(y_test), acc, 0.90, method="holdout_z_test"

)

sample = cp.estimate_sample_size(0.05, 0.90, method="holdout_z_test")

print(f"Holdout accuracy: {acc}")

print(f"90% CI: {ci}")

print(f"Test samples needed for a 0.05 radius 90% CI: {sample}")

The first three lines import the sklearn library used for classification and
the confidence-planner package. The following fragments of code load the data
into a standard pandas DataFrame object, split the data into a training and a
holdout test set, train an SVM classifier and record its classification
accuracy on the holdout. The final fragment of code performs the 90% CI
estimation and calculates the number of samples that would be needed to make
CI radius equal 0.05. Similar estimations can be performed for cross-
validation, bootstrapping, and progressive validation, by specifying a
different estimation function.

Figure 1: Confidence-planner screenshots. A) Screenshot of the confidence
planner web application. B) Graded error bars produced by the confidence-
planner package.

The same analyses can be performed online, without coding, by using the
confidence-planner web app. Figure 1 shows the CI estimation page for the
Wilson method for holdout data and graded error bars that can be created using
the python package.

##  4 Conclusions

This demo paper introduced the confidence-planner package that enables
calculating confidence bounds around classification accuracy. It provides an
easy-to-use, extensible, and freely available implementation of estimation
procedures for holdout, bootstrap, cross-validation and progressive validation
schemes. In the future we plan to extend the list of estimation methods and
provide more visualizations of uncertainty, for example for validation and
learning curves.

## References

  * [1] FDA: Software as a Medical Device (SaMD), https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd
  * [2] Blum, A., Kalai, A., Langford, J.: Beating the hold-out: Bounds for k-fold and progressive cross-validation. In: Proceedings of the Twelfth Annual Conference on Computational Learning Theory, COLT 1999. pp. 203–208 (1999)
  * [3] Clopper, C.J., Pearson, E.S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4), 404–413 (1934)
  * [4] European Commission: Proposal for a regulation of the European Parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain Union legislative acts (2021), Procedure number 2021/0106/COD
  * [5] Kovacs, M., van Ravenzwaaij, D., Hoekstra, R., Aczel, B.: SampleSizePlanner: A Tool to Estimate and Justify Sample Size for Two-Group Studies. Advances in Methods and Practices in Psychological Science 5(1) (2022)
  * [6] Langford, J.: Tutorial on practical prediction theory for classification. Journal of Machine Learning Research 6, 273–306 (2005)
  * [7] Puth, M.T., Neuhäuser, M., Ruxton, G.: On the variety of methods for calculating confidence intervals by bootstrapping. The Journal of animal ecology 84 (2015)
  * [8] Draft Recommendation on the Ethics of Artificial Intelligence. In: Report of the Social and Human Sciences Commission (SHS) Conference: UNESCO. General Conference, 41st (2021), Document code 41 C/73
  * [9] Wilson, E.B.: Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158), 209–212 (1927)


(Suman and Jain, 2021). For all other tasks, we feed the last hidden state of
the encoder corresponding to the [CLS] token into a separate
classifier/regression head consisting of two linear layers of hidden size 768
and a dropout probability of 0.1. We use the mean squared error loss for
regression tasks and the cross-entropy loss for all other tasks.

####  B.4.2 Training strategies for language model finetuning (§4, §6)

When training models for the benchmark (§4) and the multi-task (§6)
experiments, the learning rate was linearly increased for 6% of the training
steps up to 1e-5 and linearly decreased afterward. All models were trained for
a maximum of 10 epochs using three different seeds, with early stopping after
validation performance did not increase for three consecutive epochs.

Our multi-task training in §6 requires two stages of training: (1) a pre-
finetuning stage that simultaneously trains a model on multiple different
tasks, and (2) a finetuning stage that loads the model trained from (1) and
finetunes it to a single task. In the first stage, a single batch can include
several different tasks and produce different types of losses. To obtain a
unified loss that is differentiable, we aggregated the loss for each sample
and sum them up, which we use for backpropagation. For both stages, we use the
same aforementioned training steps and learning rate strategy.

For all settings, the training batch size was set to 32 with 16-bit precision
enabled. Validation was made after each training epoch on the validation set
using Pearson’s r correlation added by 1 and divided by 2 for regression tasks
and macro F1 score for all other tasks. If there were multiple tasks
considered due to multi-task training, the average of all task performances
was used as the final validation score.

###  B.5 Details on prompt-based finetuning (§4, §5)

We use fix prompts fine-tuning for all the prompt-based models. The batch size
was set as 32 for training. For every single task, we set 10 as the max epoch
and do early stopping based on the validation loss. The learning rate is set
as 5e-5.

For classification tasks, the model is fine-tuned to generate the target
label. For regression tasks, we first normalized the scores into (0,1) and
then split the labels into two groups. The model is fine-tuned to predict
“yes” or “no” regarding the prompt question. During inference, the probability
of the “yes” token is used as the prediction score. For span tasks, we
directly train the model to generate the full answer.

###  B.6 Details on zero-shot predictions (§4, §5)

We use manually designed prompts for all the zero-shot prediction tasks and
the prompts are shown in Table 8.

###  B.7 Computing correlation scores of task dependencies (§5)

Because our framework consists of several task types, it is challenging to
obtain a unified metric of correlation across different task comparisons. We
use the following rules to obtain correlation values:

  * •

Regression task & regression task: We compute the Pearson’s correlation
coefficient of the two arrays.

  * •

Regression task & binary classification task: We compute the point biserial
correlation coefficient of a continuous array and a binary array.

  * •

Regression task & multi-class classification task: We set up a linear
regression task using the one-hot coded values of the multi-class array as
independent variables and the continuous array as the dependent variable. We
report the root of the R-squared value of the regression as correlation
(Olsson et al., 1982).

  * •

Binary classification task & binary classification task: We compute the
Matthews’ correlation coefficient (Matthews, 1975) from the two binary arrays.

  * •

Binary or multi-task classification task & multi-class classification task: We
compute the Cramer’s V score (Cramér, 1999) from the two arrays of categorical
variables.

###  B.8 Computing pairwise model similarities (§5)

We quantify the model similarity between two tasks as follows. We finetune a
pretrained LLM on task $t_{A}$ to obtain a model $m_{A}$, and another LLM on
task $t_{B}$ to obtain $m_{B}$. We obtain pairwise model similarities by
inferring both models on a sufficiently large dataset—in this case the entire
test set of all tasks—and computing the correlation of the two inferred
arrays. We construct an undirected graph (Figure 7) where the thickness and
color represent absolute correlation strength and polarity between the two
models. The addition of polarity enables us to further discover strong
negative correlations with task pairs such as politeness and offensiveness.

Type | Task |  Question/Options |
---|---|---|---
PAIR | talkdown-pairs |  For the quote "text_a" and its context "text_b", is the quote condescending? |  [’No’, ’Yes’]
REG | hahackathon#humor_rating |  Determine the degree of humor of the given sentence: "text". The score should be ranging from 0.0 to 5.0, and can be a decimal. 0 is not humorous at all, and 5 is very humorous. |
CLS | hahackathon#is_humor |  For the sentence: "text", is it humorous? |  [’No’, ’Yes’]
PAIR | humor-pairs |  The first sentence is "text_a". The second sentence is "text_b". Is the first sentence funnier than the second sentence? |  [’Yes’, ’No’]
CLS | sarc |  For the sentence: "text", is it sarcastic? |  [’Yes’, ’No’]
CLS | tweet_irony |  For the sentence: "text", is it ironic? |  [’No’, ’Yes’]
CLS | contextual-abuse#IdentityDirectedAbuse |  For the sentence: "text", is it identity directed abuse? |  [’No’, ’Yes’]
CLS | contextual-abuse#PersonDirectedAbuse |  For the sentence: "text", is it person directed abuse? |  [’No’, ’Yes’]
REG | hahackathon#offense_rating |  Determine the degree of offense of the given sentence: "text". The score should be ranging from 0.0 to 5.0, and can be a decimal. |
CLS | hasbiasedimplication |  For the sentence: "text", does it imply some biases? |  [’No’, ’Yes’]
CLS | hateoffensive |  For the sentence: "text", is it hate or offensive? |  [’Hate’, ’Offensive’, ’Neither’]
CLS | implicit-hate#explicit_hate |  For the sentence: "text", is it explicit hate? |  [’No’, ’Yes’]
CLS | implicit-hate#implicit_hate |  For the sentence: "text", is it implicitly hateful? |  [’No’, ’Yes’]
CLS | implicit-hate#incitement_hate |  For the sentence: "text", is it a hateful incitement to act? |  [’No’, ’Yes’]
CLS | implicit-hate#inferiority_hate |  For the sentence: "text", is it inferiority hate? |  [’No’, ’Yes’]
CLS | implicit-hate#stereotypical_hate |  For the sentence: "text", is it a hateful message involving stereotypes? |  [’No’, ’Yes’]
CLS | implicit-hate#threatening_hate |  For the sentence: "text", is it hateful in a threatening way? |  [’No’, ’Yes’]
CLS | implicit-hate#white_grievance_hate |  For the sentence: "text", is it white grievance hate? |  [’No’, ’Yes’]
CLS | intentyn |  For the sentence: "text", is it intentional? |  [’No’, ’Yes’]
CLS | jigsaw#identity_hate |  For the sentence: "text", is it identity hate? |  [’No’, ’Yes’]
CLS | jigsaw#insult |  For the sentence: "text", is it an insult? |  [’No’, ’Yes’]
CLS | jigsaw#obscene |  For the sentence: "text", is it obscene? |  [’No’, ’Yes’]
CLS | jigsaw#severe_toxic |  For the sentence: "text", is it severely toxic? |  [’No’, ’Yes’]
CLS | jigsaw#threat |  For the sentence: "text", is it a threat? |  [’No’, ’Yes’]
CLS | jigsaw#toxic |  For the sentence: "text", is it toxic? |  [’No’, ’Yes’]
CLS | offensiveyn |  For the sentence: "text", is it offensive? |  [’No’, ’Yes’]
CLS | sexyn |  For the sentence: "text", is it sexist? |  [’No’, ’Yes’]
SPAN | toxic-span |  In the sentence: "text", which part of it can be identified as toxic? |
CLS | tweet_offensive |  For the sentence: "text", is it offensive? |  [’No’, ’Yes’]
CLS | crowdflower |  For the sentence: "text", what is its emotion? |  [’empty’, ’sadness’, ’enthusiasm’, ’neutral’, ’worry’, ’love’, ’fun’, ’hate’, ’happiness’, ’relief’, ’boredom’, ’surprise’, ’anger’]
CLS | dailydialog |  For the given conversation, "text", what is its emotion? |  [’no emotion’, ’anger’, ’disgust’, ’fear’, ’happiness’, ’sadness’, ’surprise’]
REG | emobank#arousal |  Given the VAD model of emotion, determine the degree of arousal of the given sentence: "text". The score should be ranging from 0.0 to 5.0, and can be a decimal. |
REG | emobank#dominance |  Given the VAD model of emotion, determine the degree of dominance of the given sentence: "text". The score should be ranging from 0.0 to 5.0, and can be a decimal. |
REG | emobank#valence |  Given the VAD model of emotion, determine the degree of valence of the given sentence: "text". The score should be ranging from 0.0 to 5.0, and can be a decimal. |
SPAN | emotion-span |  In the sentence: "text", which part of it expresses strong emotion? |
REG | empathy#distress |  Determine the degree of distress of the given sentence: "text". The score should be ranging from 0.0 to 7.0, and can be a decimal. |
CLS | empathy#distress_bin |  For the sentence: "text", is it showing distress? |  [’No’, ’Yes’]
PAIR | same-side-pairs |  For the sentences: "text_a" and "text_b", are they on the same side? |  [’No’, ’Yes’]
CLS | sentitreebank |  For the sentence: "text", is it positive? |  [’Yes’, ’No’]
CLS | tweet_emoji |  For the sentence: "text", what is the emoji that can be added to it? |  20 emojis
CLS | tweet_emotion |  For the sentence: "text", what is its emotion? |  [’anger’, ’joy’, ’optimism’, ’sadness’]
CLS | tweet_sentiment |  For the sentence: "text", what is its sentiment? |  [’negative’, ’neutral’, ’positive’]
CLS | complaints |  For the sentence: "text", is it a complaint? |  [’No’, ’Yes’]
REG | empathy#empathy |  Determine the degree of empathy of the given sentence: "text". The score should be ranging from 0.0 to 7.0, and can be a decimal. |
CLS | empathy#empathy_bin |  For the sentence: "text", is it expressing empathy? |  [’No’, ’Yes’]
CLS | hayati_politeness |  For the sentence: "text", is it polite? |  [’No’, ’Yes’]
CLS | questionintimacy |  For the sentence: "text", how intimate do you think it is? |  [’Very intimate’, ’Intimate’, ’Somewhat intimate’, ’Not very intimate’, ’Not intimate’, ’Not intimate at all’]
CLS | stanfordpoliteness |  For the sentence: "text", is it polite? |  [’Yes’, ’No’]
CLS | bragging#brag_achievement |  For the sentence: "text", is it bragging about an achievement? |  [’No’, ’Yes’]
CLS | bragging#brag_action |  For the sentence: "text", is it bragging about an action? |  [’No’, ’Yes’]
CLS | bragging#brag_possession |  For the sentence: "text", is it bragging about a possession? |  [’No’, ’Yes’]
CLS | bragging#brag_trait |  For the sentence: "text", is it bragging about a trait? |  [’No’, ’Yes’]
CLS | hypo-l |  For the sentence: "text", is it a hyperbole? |  [’No’, ’Yes’]
PAIR | neutralizing-bias-pairs |  For the sentences: "text_a" and "text_b", which one is biased? |  [’the first sentence is biased’, ’the second sentence is biased’]
SPAN | propaganda-span |  In the sentence: "text", which part of it can be identified as the propaganda? |
CLS | rumor#rumor_bool |  For the sentence: "text", is it a rumor? |  [’No’, ’Yes’]
CLS | two-to-lie#receiver_truth |  For the sentence :"text", will it be perceived as a lie by the receiver? |  [’Yes’, ’No’]
CLS | two-to-lie#sender_truth |  For the sentence :"text", is the sender intending to tell a lie? |  [’Yes’, ’No’]

Table 8: The manually designed prompt questions and options used for each
task.

Type | Task |  Question
---|---|---
REG | hahackathon#humor_rating |  For the sentence:"text", is it humorous?
REG | hahackathon#offense_rating |  For the sentence:"text", is it offensive?
REG | emobank#arousal |  For the sentence:"text", is the presented emotion highly arousal?
REG | emobank#dominance |  For the sentence:"text", is the presented emotion highly dominant?
REG | emobank#valence |  For the sentence:"text", is the presented emotion positive?
REG | empathy#distress |  For the sentence:"text", does it show distress?
REG | empathy#empathy |  For the sentence:"text", does it show empathy?

Table 9: The manually designed prompt questions for fine-tuning regression tasks over the t5 model. Figure 6: A detailed heatmap of Figure 2 showing task dependency among all task pairs as well as task labels. Each value represents the absolute strength of correlation between the true labels of the test set of a specific task (columns) and the predictions made on that task using a model trained on a different task (rows). Figure 7: Weighted, undirected graph of model correlations. Each edge between nodes $i$ and $j$ is weighted by the correlation between predictions from a model fine-tuned on task $i$ and predictions from a model fine-tuned on task $j$, evaluated on the entire SocKET dataset. Nodes are sized proportionally to their weighted degree and a Yifan Hu algorithm is applied for layout, with minor adjustments for readability. Refer to §B.8 for details on how the pairwise score for each edge was computed. We observe strong positive correlations similar to Figure 2, especially within the Sentiment & Emotion category and the Offensiveness category. We also see that across-category transfers may happen in a negative direction such as hayati_politeness and several Offensiveness tasks. Dataset | Task name | Task type | Rand. | Maj. | Single task | Categorywise | All tasks
---|---|---|---|---|---|---|---
Humor & Sarcasm |  |  |  |  |  |  |
hahackathon | humor_rating | REG | 0.5 | 0.5 | 0.68 0.01 | 0.67 0.01 | 0.66 0.03
hahackathon | is_humor | CLS | 0.49 | 0.38 | 0.93 0.00 | 0.93 0.00 | 0.91 0.01
humor-pairs | humor-pairs | PAIR | 0.50 | 0.34 | 0.98 0.00 | 0.98 0.00 | 0.97 0.00
tweet_irony | tweet_irony | CLS | 0.49 | 0.28 | 0.80 0.01 | 0.80 0.00 | 0.75 0.03
Offensiveness |  |  |  |  |  |  |
contextual-abuse | IdentityDirectedAbuse | CLS | 0.35 | 0.50 | 0.61 0.02 |  0.62 0.00 |  0.63 0.01
contextual-abuse | PersonDirectedAbuse | CLS | 0.34 | 0.50 | 0.54 0.01 |  0.55 0.02 |  0.57 0.02
hahackathon | offense_rating | REG | 0.5 | 0.5 | 0.91 0.01 | 0.91 0.00 |  0.92 0.01
hasbiasedimplication | hasbiasedimplication | CLS | 0.50 | 0.37 | 0.86 0.00 | 0.86 0.00 |  0.87 0.00
hateoffensive | hateoffensive | CLS | 0.27 | 0.27 | 0.93 0.00 |  0.95 0.01 |  0.95 0.01
implicit-hate | explicit_hate | CLS | 0.37 | 0.49 | 0.72 0.00 |  0.74 0.01 |  0.73 0.01
implicit-hate | implicit_hate | CLS | 0.47 | 0.43 | 0.71 0.01 | 0.71 0.01 | 0.69 0.01
implicit-hate | incitement_hate | CLS | 0.38 | 0.49 | 0.67 0.01 |  0.68 0.01 | 0.67 0.00
implicit-hate | inferiority_hate | CLS | 0.34 | 0.50 | 0.59 0.02 | 0.59 0.01 | 0.58 0.02
implicit-hate | stereotypical_hate | CLS | 0.37 | 0.49 | 0.68 0.01 | 0.66 0.01 | 0.68 0.01
implicit-hate | threatening_hate | CLS | 0.33 | 0.50 | 0.60 0.02 |  0.64 0.01 |  0.65 0.04
implicit-hate | white_grievance_hate | CLS | 0.39 | 0.48 | 0.71 0.01 | 0.71 0.01 | 0.71 0.02
intentyn | intentyn | CLS | 0.47 | 0.42 | 0.75 0.00 | 0.74 0.01 | 0.74 0.01
jigsaw | identity_hate | CLS | 0.34 | 0.50 | 0.79 0.01 | 0.76 0.01 | 0.79 0.01
jigsaw | insult | CLS | 0.38 | 0.49 | 0.88 0.00 | 0.87 0.01 | 0.88 0.01
jigsaw | obscene | CLS | 0.38 | 0.49 | 0.90 0.01 |  0.91 0.00 |  0.91 0.01
jigsaw | severe_toxic | CLS | 0.34 | 0.50 | 0.74 0.01 | 0.73 0.02 | 0.71 0.02
jigsaw | threat | CLS | 0.34 | 0.50 | 0.85 0.02 | 0.81 0.04 | 0.80 0.03
jigsaw | toxic | CLS | 0.41 | 0.47 | 0.91 0.00 | 0.90 0.00 | 0.91 0.01
offensiveyn | offensiveyn | CLS | 0.50 | 0.37 | 0.82 0.00 |  0.83 0.01 |  0.83 0.00
sexyn | sexyn | CLS | 0.37 | 0.49 | 0.80 0.01 | 0.79 0.00 | 0.79 0.00
talkdown-pairs | talkdown-pairs | PAIR | 0.50 | 0.33 | 0.89 0.01 | 0.88 0.01 | 0.88 0.00
toxic-span | toxic-span | SPAN | 0 | 0 | 0.68 0.02 | 0.65 0.05 | 0.67 0.05
tweet_offensive | tweet_offensive | CLS | 0.48 | 0.42 | 0.81 0.01 | 0.81 0.01 | 0.80 0.00
Sentiment & Emotion |  |  |  |  |  |  |
crowdflower | crowdflower | CLS | 0.06 | 0.03 | 0.24 0.00 | 0.23 0.01 | 0.22 0.00
dailydialog | dailydialog | CLS | 0.07 | 0.13 | 0.43 0.04 |  0.47 0.00 |  0.47 0.02
emobank | arousal | REG | 0.5 | 0.5 | 0.80 0.02 |  0.81 0.02 | 0.79 0.02
emobank | dominance | REG | 0.5 | 0.5 | 0.75 0.02 | 0.75 0.01 | 0.73 0.02
emobank | valence | REG | 0.5 | 0.5 | 0.92 0.01 | 0.92 0.00 | 0.90 0.01
emotion-span | emotion-span | SPAN | 0 | 0 | 0.96 0.01 | 0.89 0.00 | 0.86 0.03
empathy | distress | REG | 0.5 | 0.5 | 0.77 0.01 | 0.75 0.03 | 0.72 0.01
empathy | distress_bin | CLS | 0.50 | 0.31 | 0.68 0.01 |  0.69 0.03 | 0.65 0.02
same-side-pairs | same-side-pairs | PAIR | 0.48 | 0.35 | 0.66 0.12 |  0.70 0.09 |  0.76 0.05
sentitreebank | sentitreebank | CLS | 0.50 | 0.31 | 0.97 0.00 | 0.96 0.00 | 0.96 0.01
tweet_emoji | tweet_emoji | CLS | 0.04 | 0.02 | 0.34 0.00 | 0.33 0.00 | 0.33 0.00
tweet_emotion | tweet_emotion | CLS | 0.24 | 0.14 | 0.80 0.01 |  0.81 0.01 | 0.80 0.00
tweet_sentiment | tweet_sentiment | CLS | 0.32 | 0.22 | 0.71 0.00 | 0.71 0.01 | 0.69 0.01
Social Factors |  |  |  |  |  |  |
complaints | complaints | CLS | 0.50 | 0.36 | 0.92 0.01 | 0.92 0.00 | 0.91 0.01
empathy | empathy | REG | 0.5 | 0.5 | 0.70 0.04 |  0.71 0.03 | 0.70 0.01
empathy | empathy_bin | CLS | 0.50 | 0.33 | 0.63 0.02 | 0.62 0.01 | 0.59 0.03
hayati_politeness | hayati_politeness | CLS | 0.47 | 0.41 | 0.87 0.01 | 0.87 0.05 |  0.89 0.03
questionintimacy | questionintimacy | CLS | 0.16 | 0.06 | 0.49 0.03 | 0.48 0.02 | 0.46 0.02
stanfordpoliteness | stanfordpoliteness | CLS | 0.50 | 0.36 | 0.70 0.02 |  0.71 0.01 |  0.72 0.02
Trustworthiness |  |  |  |  |  |  |
bragging | brag_achievement | CLS | 0.36 | 0.49 | 0.74 0.01 | 0.69 0.01 |  0.76 0.03
bragging | brag_action | CLS | 0.35 | 0.50 | 0.59 0.02 | 0.57 0.06 | 0.59 0.04
bragging | brag_possession | CLS | 0.35 | 0.50 | 0.70 0.02 | 0.66 0.03 | 0.52 0.03
bragging | brag_trait | CLS | 0.34 | 0.50 | 0.67 0.01 | 0.59 0.07 | 0.61 0.04
hypo-l | hypo-l | CLS | 0.48 | 0.41 | 0.74 0.01 | 0.71 0.01 | 0.69 0.01
neutralizing-bias-pairs | neutralizing-bias-pairs | PAIR | 0.50 | 0.33 | 0.96 0.01 | 0.96 0.01 | 0.96 0.00
propaganda-span | propaganda-span | SPAN | 0 | 0 | 0.22 0.10 |  0.23 0.03 |  0.24 0.00
rumor | rumor_bool | CLS | 0.49 | 0.39 | 0.85 0.05 | 0.78 0.02 | 0.78 0.05
two-to-lie | receiver_truth | CLS | 0.38 | 0.49 | 0.57 0.02 | 0.57 0.02 | 0.53 0.01
two-to-lie | sender_truth | CLS | 0.38 | 0.49 | 0.58 0.02 |  0.59 0.03 | 0.55 0.03
Table 10: Detailed table of performance scores from comparing single-task vs
multi-task trained models in Section 6 (refer to Table 3 in Section 6). There
are no significant gains from the two multi-task settings in the Humor &
Sarcasm category, where the tasks in general have low task dependency (ref.
Section 5). However, for other categories we see several instances of tasks
where multi-task trained model have greater performance.

###  B.9 List of all potential tasks and datasets for SocKET

Table 11: Table of all the datasets considered when curating the SocKET Benchmark. Paper/Dataset Title |  Tasks |  Reference
---|---|---
Automatic Identification and Classification of Bragging in Social Media |  Bragging (Achievement) |  Jin et al. (2022)
Automatic Identification and Classification of Bragging in Social Media Jin et al. (2022) |  Bragging (Action) |  Jin et al. (2022)
Automatic Identification and Classification of Bragging in Social Media |  Bragging (Possession) |  Jin et al. (2022)
Automatic Identification and Classification of Bragging in Social Media |  Bragging (Trait) |  Jin et al. (2022)
Automatically Identifying Complaints in Social Media |  Complaints |  Preoţiuc-Pietro et al. (2019)
Introducing CAD: the Contextual Abuse Dataset |  Identity Based Hate |  Vidgen et al. (2021)
Introducing CAD: the Contextual Abuse Dataset |  Individual Hate |  Vidgen et al. (2021)
Introducing CAD: the Contextual Abuse Dataset |  Group-Based Hate |  Vidgen et al. (2021)
Introducing CAD: the Contextual Abuse Dataset |  Counter Speech |  Vidgen et al. (2021)
Sentiment Analysis in Text |  Emotion |  CrowdFlower (2016)
DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset |  Emotion |  Li et al. (2017)
EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis |  Emotion (Valence) |  Buechel and Hahn (2017)
EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis |  Emotion (Arousal) |  Buechel and Hahn (2017)
EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis |  Emotion (Dominance) |  Buechel and Hahn (2017)
Detecting Emotion Stimuli in Emotion-Bearing Sentences |  Emotion |  Ghazi et al. (2015)
Measuring the Language of Self-Disclosure across Corpora |  Disturbance |  Reuel et al. (2022)
Measuring the Language of Self-Disclosure across Corpora |  Empathy |  Reuel et al. (2022)
SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense |  Humor Rating |  Meaney et al. (2021)
SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense |  Funny (boolean) |  Meaney et al. (2021)
SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense |  Offensiveness |  Meaney et al. (2021)
Social Bias Frames: Reasoning about Social and Power Implications of Language |  Biased Implication |  Sap et al. (2020)
Social Bias Frames: Reasoning about Social and Power Implications of Language |  Intent |  Sap et al. (2020)
Social Bias Frames: Reasoning about Social and Power Implications of Language |  Offensiveness |  Sap et al. (2020)
Social Bias Frames: Reasoning about Social and Power Implications of Language |  Sexism |  Sap et al. (2020)
Automated Hate Speech Detection and the Problem of Offensive Language |  Offensive |  Davidson et al. (2017)
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica |  Politeness |  Hayati et al. (2021)
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica |  Positivity |  Hayati et al. (2021)
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica |  Anger |  Hayati et al. (2021)
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica |  Disgust |  Hayati et al. (2021)
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica |  Fear |  Hayati et al. (2021)
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica |  Joy |  Hayati et al. (2021)
Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica |  Sadness |  Hayati et al. (2021)
SemEval-2020 Task 7: Assessing Humor in Edited News Headlines |  Funnier Sequence |  Hossain et al. (2020)
MOVER: Mask, Over-generate and Rank for Hyperbole Generation |  Hyperbole |  Zhang and Wan (2022)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Explicit Hate |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Implicit Hate |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Incitement |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Inferiority |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Stereotyping |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Threat |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Offensive |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Irony |  ElSherief et al. (2021)
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech |  Other Hate |  ElSherief et al. (2021)
Toxic Comment Classification Challenge |  Identity-Based Hate |  Jigsaw (2017)
Toxic Comment Classification Challenge |  Insult |  Jigsaw (2017)
Toxic Comment Classification Challenge |  Obscenity |  Jigsaw (2017)
Toxic Comment Classification Challenge |  Severe Toxicity |  Jigsaw (2017)
Toxic Comment Classification Challenge |  Threat |  Jigsaw (2017)
Toxic Comment Classification Challenge |  Toxicity |  Jigsaw (2017)
Automatically Neutralizing Subjective Bias in Text |  Bias |  Pryzant et al. (2020)
SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles |  Propaganda Technique |  Da San Martino et al. (2020)
Quantifying Intimacy in Language |  Intimacy |  Pei and Jurgens (2020)
Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning |  Rumor Detection |  Ma et al. (2017)
On Classifying whether Two Texts are on the Same Side of an Argument |  Stance |  Körner et al. (2021)
A Large Self-Annotated Corpus for Sarcasm |  Sarcasm |  Khodak et al. (2018)
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank |  Sentiment |  Socher et al. (2013)
Facilitating the Communication of Politeness through Fine-Grained Paraphrasing |  Politeness |  Fu et al. (2020)
TalkDown: A Corpus for Condescension Detection in Context |  Condescension |  Wang and Potts (2019)
SemEval-2021 Task 5: Toxic Spans Detection |  Toxicity |  Pavlopoulos et al. (2021)
SemEval 2018 Task 2: Multilingual Emoji Prediction |  Emoji |  Barbieri et al. (2018)
SemEval-2018 Task 1: Affect in Tweets |  Emotion |  Mohammad et al. (2018)
SemEval-2018 Task 3: Irony Detection in English Tweets |  Irony |  Van Hee et al. (2018)
Predicting the Type and Target of Offensive Posts in Social Media |  Offensiveness |  Zampieri et al. (2019a)
SemEval-2017 Task 4: Sentiment Analysis in Twitter |  Sentiment |  Rosenthal et al. (2017)
It Takes Two to Lie: One to Lie, and One to Listen |  Sender Truth |  Peskov et al. (2020)
It Takes Two to Lie: One to Lie, and One to Listen |  Receiver Truth |  Peskov et al. (2020)
“So You Think You’re Funny?”: Rating the Humour Quotient in Standup Comedy |  Humor Rating |  Mittal et al. (2021)
DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates |  Stance |  Pougué-Biyong et al. (2021)
The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News |  Trustworthiness |  Nakov et al. (2021)
Finding Deceptive Opinion Spam by Any Stretch of the Imagination |  Deceipt |  Ott et al. (2011)
Finding Deceptive Opinion Spam by Any Stretch of the Imagination |  Fact |  Ott et al. (2011)
A Clustering Approach for Nearly Unsupervised Recognition of Nonliteral Language |  Nonliteral Langauge |  Birke and Sarkar (2006)
Detecting Community Sensitive Norm Violations in Online Conversations |  Community Norms |  Park et al. (2021)
Can Machines Learn Morality? The Delphi Experiment |  Moral Judgement |  Jiang et al. (2021)
SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter |  Hate Speech |  Basile et al. (2019)
SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) |  Offensiveness |  Zampieri et al. (2019b)
CivilComments |  Toxicity |  Jigsaw (2019)
CivilComments |  Very Toxic |  Jigsaw (2019)
(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas |  Gender |  Kang et al. (2019)
(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas |  Age |  Kang et al. (2019)
(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas |  Country |  Kang et al. (2019)
(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas |  Political view |  Kang et al. (2019)
(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas |  Education |  Kang et al. (2019)
(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas |  Ethnicity |  Kang et al. (2019)
Webis Clickbait Corbus 2017 |  Clickbait |  Potthast et al. (2018)
VU Amsterdam Metaphor Corpus |  Metaphor |  Steen et al. (2011)
Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications |  Uncertainty |  Pei and Jurgens (2021)
Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer |  Formality |  Rao and Tetreault (2018)
International Survey on Emotion Antecedents and Reactions |  Sentiment |  Scherer and Wallbott (1994)
Short Jokes |  Joke |  Moudgil
Short Text Corpus with Focus on Humor Detection |  Joke |  CrowdTruth (2016)
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter |  Sexism |  Waseem and Hovy (2016)
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter |  Racism |  Waseem and Hovy (2016)
Studying the Dark Triad of Personality through Twitter Behavior |  narcissism |  Preoţiuc-Pietro et al. (2019)
Studying the Dark Triad of Personality through Twitter Behavior |  psychopathy |  Preoţiuc-Pietro et al. (2019)
Studying the Dark Triad of Personality through Twitter Behavior |  Machiavellianism |  Preoţiuc-Pietro et al. (2019)
Utterance-level Dialogue Understanding: An Empirical Study |  Emotion |  Ghosal et al. (2020)