# LLMCheckup:
Conversational Examination of Large Language Models
via Interpretability Tools

Qianli Wang1,2 Tatiana Anikina∗1,3 Nils Feldhus∗1
Josef van Genabith1,3 Leonhard Hennig1 Sebastian Möller1,2
1German Research Center for Artificial Intelligence (DFKI)
2Technische Universität Berlin, Germany
3Saarland Informatics Campus, Saarbrücken, Germany
<EMAIL_ADDRESS>

###### Abstract

Interpretability tools that offer explanations in the form of a dialogue have
demonstrated their efficacy in enhancing users’ understanding Slack et al.
(2023); Shen et al. (2023), as one-off explanations may occasionally fall
short in providing sufficient information to the user. Current solutions for
dialogue-based explanations, however, require many dependencies and are not
easily transferable to tasks they were not designed for. With
LLMCheckup111Code: https://github.com/DFKI-NLP/LLMCheckup Demo:
https://youtu.be/ZwN8ZQSXoOU, we present an easily accessible tool that allows
users to chat with any state-of-the-art large language model (LLM) about its
behavior. We enable LLMs to generate all explanations by themselves and take
care of intent recognition without fine-tuning, by connecting them with a
broad spectrum of Explainable AI (XAI) tools, e.g. feature attributions,
embedding-based similarity, and prompting strategies for counterfactual and
rationale generation. LLM (self-) explanations are presented as an interactive
dialogue that supports follow-up questions and generates suggestions.
LLMCheckup provides tutorials for operations available in the system, catering
to individuals with varying levels of expertise in XAI and supports multiple
input modalities. We introduce a new parsing strategy called multi-prompt
parsing substantially enhancing the parsing accuracy of LLMs. Finally, we
showcase the tasks of fact checking and commonsense question answering.

LLMCheckup:
Conversational Examination of Large Language Models
via Interpretability Tools

Qianli Wang1,2 Tatiana Anikina∗1,3 Nils Feldhus∗1 Josef van Genabith1,3
Leonhard Hennig1 Sebastian Möller1,2 1German Research Center for Artificial
Intelligence (DFKI) 2Technische Universität Berlin, Germany 3Saarland
Informatics Campus, Saarbrücken, Germany<EMAIL_ADDRESS>

††*Equal contribution

Figure 1: LLMCheckup dialogue with data augmentation and rationalization
operations on a commonsense question answering task (ECQA). Boxes (not part of
the actual UI) indicate the original instance from the dataset as well as its
prediction (cyan) and the explanation requested by the user (orange).

##  1 Introduction

To unravel the black box nature of deep learning models for natural language
processing, a diverse range of explainability methods have been developed
Ribeiro et al. (2016); Madsen et al. (2022); Wiegreffe et al. (2022).
Nevertheless, practitioners often face difficulties in effectively utilizing
explainability methods, as they may not be aware of which techniques are
available or how to interpret results provided. There has been a consensus
within the research community that moving beyond one-off explanations and
embracing conversations to provide explanations is more effective for model
understanding Lakkaraju et al. (2022); Feldhus et al. (2023); Zhang et al.
(2023) and helps mitigate the limitations associated with the effective usage
of explainability methods to some extent Ferreira and Monteiro (2020); Slack
et al. (2023).

Figure 2:  On the left, a dialogue example asking for explanation in natural
language about a ECQA-like customized question. The workflow of LLMCheckup is
shown on the right side.

In the field of NLP, two dialogue-based interpretability tools, InterroLang
Feldhus et al. (2023) and ConvXAI Shen et al. (2023), have been introduced.
Both tools employ multiple, separately fine-tuned LMs to parse user intents
and dedicated external LMs to provide explanations.

By contrast, our system, LLMCheckup, only requires a single LLM and puts it on
“quadruple duty”: (1) Analyzing users’ (explanation) requests (§2.1, §5.2),
(2) performing downstream tasks (§4), (3) providing explanations for its
outputs (§3), and (4) responding to the users in natural language (§2.3).
Instead of using many LMs to explain the behavior of another LLM, LLMCheckup
allows us to directly request the same LLM used for parsing to self-explain
its own behavior. Any LLM can be deployed “out of the box” and confronted with
diverse NLP tasks, explanation generation and intent recognition. LLMCheckup
provides a unified and compact framework, effectively consolidating and
obviating the requirements for additional LMs to generate explanations.

##  2 LLMCheckup

LLMCheckup is an interface for chatting with any LLM about its behavior. We
connect several interpretability tools (§3), s.t. self-explanations take model
internals, datasets and documentation into account. User requests for
explanations are recognized via a text-to-SQL task performed by the LLM
(§2.1).

###  2.1 System architecture

Figure 2 illustrates the interaction flow of LLMCheckup. When a user asks a
question on the user interface, the question will be parsed as an SQL-like
query by the LLM in use. E.g., the first user question in Figure 1 would be
parsed as "filter id 26 and rationalize". The corresponding operation (see
Table 6) will then be matched (parsing) and executed. Finally, the generated
explanation will be filled into a template that will be returned back to the
user Slack et al. (2023); Feldhus et al. (2023) (response generation).

#### Parsing

To enhance the recognition of users’ intentions, we employ two methods: Guided
Decoding and Multi-prompt Parsing.

Guided Decoding (GD) ensures that the generated output adheres to predefined
grammatical rules and constraints Shin et al. (2021) and that generated parses
align with predefined operation sets Slack et al. (2023). GD is generally more
suitable for smaller LMs.

As an alternative to GD, we also propose and implement a novel Multi-prompt
Parsing (MP) approach. While GD pre-selects prompts based on the embedding
similarity with user input, the model does not see all the available
operations at once and the pre-selection may not include the examples for the
actual operation required. With MP, we test whether showing all possible
operations in a simplified format as the first step (i.e., without any
attributes such as the instance ID or the number of samples) and then
additionally prompting the model to fill in more fine-grained attributes can
improve performance. We provide further details about MP in Appendix B. §5.2
evaluates the performance of both parsing approaches.

###  2.2 Key features

#### Supported NLP models

We identify five auto-regressive LLMs representative of the current state-of-
the-art in open-source NLP (as indicated in the left column of Table 2)
available through Hugging Face Transformers Wolf et al. (2020). The diverse
choice of models demonstrates that our system is generalizable and that in
principle it is theoretically possible to choose any Transformer-type model to
be explained. While Falcon-1B Penedo et al. (2023) and Pythia-2.8B Biderman et
al. (2023) are available for users with limited hardware resources (RAM/GPU),
it is generally not recommended to use them due to their small model size,
which may negatively affect the user’s perception of the performance.
Llama2-7B Touvron et al. (2023) and Mistral-7B Jiang et al. (2023) are both
mid-sized with 7B parameters, while Stable Beluga 2 Mukherjee et al. (2023) is
a fine-tuned version of Llama2-70B. To facilitate the deployment of large
models in a local environment, LLMCheckup offers support for various forms of
LLMs. This includes the usage of quantized models through GPTQ Frantar et al.
(2023), loading models in 4-bits with the assistance of bitsandbytes Dettmers
et al. (2022), and the implementation of a peer-to-peer solution using Petals
Borzunov et al. (2023), enabling efficient deployment on a custom-level GPU.

#### Tutorial

To help laypeople get background knowledge in XAI, we introduce a tutorial
functionality. It is based on prompting with different roles corresponding to
levels of expertise in XAI (Figure 3) and enables us to provide tailored meta-
explanations of supported operations to individuals. For example, for the
beginner level, we add a system prompt hinting at the expertise: “As a NLP
beginner, could you explain me the question: What is data augmentation?”
(Figure 4).

#### External information retrieval

LLMs may sometimes generate responses that appear plausible but are actually
incorrect predictions known as hallucinations Welleck et al. (2020).
LLMCheckup provides users with the capability to access information by
conducting search through external knowledge bases, promoted by the
integration of googlesearch222https://github.com/Nv7-GitHub/googlesearch
(Figure 5). Users can cross-reference the retrieved information with the
provided explanations, thereby achieving a more comprehensive understanding.

#### Customized inputs & prompts

In comparison to TalkToModel Slack et al. (2023), which was limited to three
datasets, LLMCheckup offers users the freedom to enter custom inputs according
to their preferences, going beyond just querying instances from the provided
dataset. In addition, inspired by PromptSource Bach et al. (2022), users can
insert pre-defined prompts to the explanation step (§3.2), but the Prompt
Editor also allows fully customized prompts, allowing for a personalized and
tailored experience. All custom inputs will be saved and can be inspected and
reused later on via a dedicated custom input history viewer.

#### Multi-modal input format

Motivated by Malandri et al. (2023), LLMCheckup not only accepts text input
from users but also provides support for other modalities such as images and
audio. To facilitate this, we employ specific packages and models tailored to
each modality. For optical character recognition (OCR), we use
EasyOCR333https://github.com/JaidedAI/EasyOCR. For audio recognition, we
employ a lightweight fairseq S2T444https://huggingface.co/facebook/s2t-small-
librispeech-asr model Wang et al. (2020) trained on Automatic Speech
Recognition (ASR).

#### Suggestion of follow-up questions

To guide the user through the conversation, we implemented a suggestions mode.
When activated, the user will get suggestions for related operations that
LLMCheckup can perform (see Appendix C for more detail), e.g., if the user
asks about the top $k$ attributed tokens for a specific sample, they will get
a suggestion to have a look at the rationales generated by the model since
both token attributions and rationalizations belong to the “Explanation"
category also displayed in the user interface. Additionally, for each
generated suggestion we check whether it already appears in the dialogue
history to make sure that the user does not receive repetitive suggestions for
the operations that they have already been performed. E.g., if the user
inquires about the counterfactual operation and the model explains how it
works, LLMCheckup will store this information and will not suggest explaining
counterfactuals again.

#### Dialogue sharing

LLMCheckup offers the functionality to export the dialogue history between the
user and the deployed LLM as a JSON file that contains the user’s questions
and the corresponding generated responses. This simplifies data collection and
sharing of conversation logs between users.

###  2.3 Interface

LLMCheckup provides a user interface including a chat window to enter
questions and settings and is implemented in Flask and can be run as a Docker
container. LLMCheckup builds on top of the chat window from TalkToModel Slack
et al. (2023) for continuous interaction as well as buttons for question
generation and the dataset viewer from InterroLang Feldhus et al. (2023),
which facilitates dataset exploration and generates specific questions for all
available operations to inspire users to come up with their own questions.
Figure 3 shows the LLMCheckup interface.

In comparison to TalkToModel Slack et al. (2023) and InterroLang Feldhus et
al. (2023), our interface provides the following new components:

  * •

Checkbox for selecting XAI user expertise level;

  * •

Input field for custom inputs;

  * •

Prompt editor including a checkbox for selecting additional prompts and a
display of the currently used system prompt;

  * •

Support for multimodal input through buttons for image upload and audio
recording.

Dataset | COVID-Fact | ECQA
---|---|---
Model | Size | Consistency | Claim Fluency | Evidence Fluency | Consistency | Question Fluency
Mistral | 7B | 0.66 | 0.88 | 0.96 | 0.50 | 0.76
Llama2 | 7B | 0.65 | 0.88 | 0.94 | 0.50 | 0.76
Stable Beluga 2 | 70B | 1.00 | 0.85 | 0.96 | 1.00 | 0.73

Table 1:  Consistency and fluency scores of data augmentation from three
models. falcon and pythia are not considered due to poor performance because
of small model size.

##  3 NLP explainability tools

Table 3 and Table 4 show examples of explanations for each supported
explainability method by LLMCheckup.

###  3.1 White-box

We provide white-box explainability operations that are gradient-based or
embedding-related, both of which are not easily applicable to the prompting
paradigm of LLMs.

#### Feature attribution

Feature attribution methods quantify the contribution of each input token
towards the final outcome. In LLMCheckup, we deploy various auto-regressive
models (§2.2), for which Inseq Sarti et al. (2023) is used to determine
attribution scores. We support representative methods from Inseq, including
Input x Gradient Simonyan et al. (2014), Attention Bahdanau et al. (2015),
LIME Ribeiro et al. (2016), and Integrated Gradients Sundararajan et al.
(2017)555Details on the Inseq integration are described in App. E..

#### Embedding analysis

By calculating the cosine similarity between the sentence embeddings of the
instances in datasets, we can retrieve relevant examples Cer et al. (2017);
Reimers and Gurevych (2019) and present them for contextualizing the model
behavior on the input in question.

###  3.2 Black-box

#### Data augmentation

Augmentation involves synthesizing new instances by replacing text spans of
the input while preserving the semantic meaning and predicted outcomes Ross et
al. (2022). As a baseline, NLPAug is used to substitute input by WordNet’s
Miller (1995) synonyms666https://github.com/makcedward/nlpaug NLPAug offers
various techniques for generating augmented text, including back-translation,
word embedding substitution, etc.. However, we constrain the augmentation to
only utilize WordNet to avoid additional LMs.. Alternatively, data
augmentation can be achieved by providing the LLM with a few demonstrations as
prompts Dai et al. (2023).

#### Counterfactual generation

Unlike data augmentation, counterfactuals are about input edits causing the
predicted outcome to be different Wu et al. (2021); Chen et al. (2023).
Counterfactuals are generated by prompting LLMs with manually crafted
demonstrations.

#### Rationalization

Rationalization aims to provide free-text explanations that elucidate the
prediction made by the model Camburu et al. (2018); Wiegreffe et al. (2022)
(an example is shown in Figure 1). The use of Chain-of-Thought (CoT) prompting
enhances the reasoning capabilities of LLMs by encouraging the generation of
intermediate reasoning steps that lead to a final answer Wei et al. (2022);
Wang et al. (2023b). Different CoT strategies can be applied depending on
users’ preferences, including Zero-CoT Kojima et al. (2022), Plan-and-Solve
Wang et al. (2023a), and Optimization by PROmpting (OPRO) Yang et al. (2023)
(Figure 3).

##  4 Use cases

In this paper, we demonstrate the usefulness of LLMCheckup with two typical
NLP tasks: Fact checking and commonsense question answering.

###  4.1 Fact checking

The importance of fact checking has grown significantly due to the rapid
dissemination of both accurate information and misinformation within the
modern media ecosystem Guo et al. (2022). COVID-Fact Saakyan et al. (2021) is
a dataset resembling FEVER Thorne et al. (2018) that encompasses various
claims, supporting evidence for those claims, and contradictory claims that
have been debunked by the presented evidence.

###  4.2 Commonsense question answering

Unlike question answering, commonsense question answering (CQA) involves the
utilization of background knowledge that may not be explicitly provided in the
given context Ostermann et al. (2018). The challenge lies in effectively
integrating a system’s comprehension of commonsense knowledge and leveraging
it to provide accurate responses to questions. ECQA Aggarwal et al. (2021) is
a dataset designed for CQA. Each instance in ECQA consists of a question,
multiple answer choices, and a range of explanations. Positive explanations
aim to provide support for the correct choice, while negative explanations
serve to refute incorrect choices. Additionally, free-text explanations in
natural language are included as general justifications.

##  5 Evaluation

###  5.1 Data augmentation

We assess the quality of the generated augmented output based on two key
aspects: (1) consistency: the metric represents the proportion of instances
where the augmentation process does not lead to a change in the label Li et
al. (2023a); Dai et al. (2023); (2) fluency: assesses how well the augmented
output aligns with the original data in terms of semantic similarity Ross et
al. (2021) measured by SBERT777https://huggingface.co/sentence-
transformers/all-mpnet-base-v2. Table 1 indicates that Mistral and Llama2
exhibit comparable performance, while Stable Beluga 2 displays substantially
higher consistency scores on two tasks, although it may exhibit lower fluency
in certain cases. The overall performance on ECQA is relatively lower compared
to COVID-Fact. This difference in performance can be attributed to the
increased complexity and difficulty of the ECQA task.

###  5.2 Parsing

To assess the ability of interpreting user intentions by LLMs, we quantify the
performance of each deployed model by calculating the exact match parsing
accuracy Talmor et al. (2017); Yu et al. (2018) on a manually created test
set, which consists of a total of 119 pairs of user questions and
corresponding SQL-like queries. As an additional baseline, we employ the
nearest neighbor approach that relies on comparing semantic similarity.

We assess parsing accuracy of our two approaches, GD and MP (§2.1). Table 2
shows that, as model size increases, the parsing accuracy tends to increase
and MP demonstrates a notable improvement over GD. Despite Stable Beluga 2
having a larger size compared to 7B models, its parsing performance only
marginally surpasses that of Mistral and Llama2. This can be partially
attributed to the difficulty of the parsing task888We have a total of 21
operations displayed in Table 6, and many of these offer multiple options. For
instance, score operation supports $F_{1}$, precision, recall and accuracy
matrices. and the number of demonstrations, as larger models may require a
greater number of demonstrations to fully comprehend the context Li et al.
(2023b). Parsing results are discussed in more detail in Appendix B.

Model | Size | Strategy | Accuracy
---|---|---|---
Nearest Neighbor | - | - | 42.24
Falcon | 1B | GD | 47.41
Pythia | 2.8B | GD | 51.72
Llama2 | 7B | GD | 64.71
Mistral | 7B | GD | 55.88
Stable Beluga 2 | 70B | GD | 67.23
Falcon | 1B | MP | 64.15
Pythia | 2.8B | MP | 75.91
Llama2 | 7B | MP | 82.35
Mistral | 7B | MP | 84.87
Stable Beluga 2 | 70B | MP | 88.24

Table 2:  Exact match parsing accuracy (in %) for different models. GD =
Guided Decoding prompted by 20-shots; MP = Multi-Prompt parsing.

##  6 Discussion

We highlight the broad generalizability of LLMCheckup concerning diverse NLP
tasks and models. Our approach empowers any auto-regressive LLM to handle
dialogue-based explainability including parsing and explanation generation.
Aside from the NLP tasks showcased in Section 4, LLMCheckup can be easily
expanded to accommodate other tasks not directly related to classification,
such as summarization or translation999For other NLP tasks, the predict
operation should be replaced by a task-specific operation and all explanation
operations can be retained and easily adapted.. In contrast to ConvXAI Shen et
al. (2023) and InterroLang Feldhus et al. (2023), which require a fine-tuned
model for each specific use case, LLMs used in LLMCheckup possess remarkable
zero-/few-shot capabilities Brown et al. (2020), which allows them to
effectively handle many tasks without requiring fine-tuning. Although the
quality of an explanation could be enhanced with further fine-tuning,
LLMCheckup uses model outputs out of the box.

##  7 Related work

#### Interfaces for interactive explanations

LIT Tenney et al. (2020) is a GUI-based tool available for analyzing model
behaviors across entire datasets. However, LIT has less functionalities in
terms of prompting and lower accessibility, e.g. no tutorial and a lower level
of integration with HuggingFace. CrossCheck Arendt et al. (2021) exhibits the
capability to facilitate quick cross-model comparison and error analysis
across various data types, but adapting it for other use cases may need a
substantial amount of code modification and customization. XMD’s Lee et al.
(2023) primary purpose is model debugging, but it shares similarities in the
focus on feature attributions, visualization of single instances and user
feedback options. It is, however, limited to feature attribution explanations
and smaller, efficiently retrainable models. IFAN Mosca et al. (2023) enables
real-time explanation-based interaction with NLP models, but is limited to the
sequence-to-class format, restricting its applicability to other tasks.
Furthermore, IFAN offers a limited set of explainability methods, consisting
of only one local and one global method.

#### Dialogue-based systems for interpretability

Carneiro et al. (2021) point out that conversational interfaces have the
potential to greatly enhance the transparency and the level of trust that
human decision-makers place in them. According to Zhang et al.’s user studies,
delivering explanations in a conversational manner can improve users’
understanding, satisfaction, and acceptance. Jacovi et al. (2023) emphasizes
the necessity of interactive interrogation in order to build understandable
explanation narratives. ConvXAI Shen et al. (2023), TalkToModel Slack et al.
(2023), InterroLang Feldhus et al. (2023) and Brachman et al. (2023) share
some similarities with our system, but are more complex in their setup and
consider fewer explainability methods. Additionally, they might overrely on
external LMs to explain the deployed model’s behavior, whereas LLMCheckup
places a strong emphasis on self-explanation, which is crucial for
faithfulness. Finally, LLMCheckup uses auto-regressive models, as they have
become increasingly dominant in various NLP applications nowadays. In iSee
Wijekoon et al. (2023), a chatbot adapts explanations to the user’s persona.
Explanation strategies and user experiences can be captured and shared. The
system is currently still in development, but shares many functionalities,
except that LLMs are not in their scope yet.

##  8 Conclusion

We present the interpretability tool LLMCheckup, designed as a dialogue-based
system. LLMCheckup can provide explanations in a conversation with the user
facilitated by any auto-regressive LLM. By consolidating parsing, downstream
task prediction, explanation generation and response generation within a
unified framework, LLMCheckup streamlines the interpretability process and
serves as a baseline for future investigation.

Future work includes exploring RAG models Lewis et al. (2020) combined with
explainability, as currently LLMCheckup relies on search engines for external
information retrieval. We also want to add multi-modal models, so converting
image or audio input to texts would no longer be necessary, but the current
state of interpretability on such models lags behind unimodal approaches Liang
et al. (2023). Integrating our system into
HuggingChat101010https://huggingface.co/chat/ would further increase the
visibility and accessibility through the web.

## Limitations

In LLMCheckup, we do not focus on dataset analysis or data-centric
interpretability, but on how a model responds to single inputs. There are a
lot of practical cases, e.g. medical report generation Messina et al. (2022),
gender-aware translation Attanasio et al. (2023), where users are not
interested in raw performance metrics on standard benchmarks, but are
interested in detecting edge cases and investigating a model’s behavior on
custom inputs.

English is the main language of the current system. Multilinguality is not
supported, as both the interface, the responses, tutorial and the explained
models are monolingual. While it would be possible to adapt it to other
languages by translating interface texts and prompts and using a model trained
on data in another target language or multiple ones, it remains to be seen to
which extent multilingual LLMs can do quadruple duty as well as the current
model does for English.

In LLMCheckup, users have the flexibility to input data in different
modalities, including images and audio. However, for audio and images,
LLMCheckup will convert the audio content and texts contained within the
images into textual format for further processing and analysis. Besides, the
explanations and responses generated by our system are currently limited to
the text format – apart from the heatmap visualization of feature attribution
explanations.

The QA tutorial only aims to provide explanations for supported operations in
XAI to individuals with different levels of expertise. However, the
explanations, e.g. rationales, generated by the LLM may not inherently adapt
to users’ specific expertise levels Zhang et al. (2023). In the future, we
will explore how to prompt the models to provide simple explanations reliably.

In LLMCheckup, we employ a single LLM to serve quadruple-duty simultaneously.
However, models with lower parameter counts may exhibit limitations in certain
types of explanation generation, particularly when using prompting techniques
like rationalization or counterfactual generation Marasovic et al. (2022).

## References

  * Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021.  Explanations for CommonsenseQA: New Dataset and Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3050–3065, Online. Association for Computational Linguistics.
  * Arendt et al. (2021) Dustin Arendt, Zhuanyi Shaw, Prasha Shrestha, Ellyn Ayton, Maria Glenski, and Svitlana Volkova. 2021.  CrossCheck: Rapid, reproducible, and interpretable model evaluation.  In _Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances_ , pages 79–85, Online. Association for Computational Linguistics.
  * Attanasio et al. (2023) Giuseppe Attanasio, Flor Plaza del Arco, Debora Nozza, and Anne Lauscher. 2023.  A tale of pronouns: Interpretability informs gender bias mitigation for fairer instruction-tuned machine translation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3996–4014, Singapore. Association for Computational Linguistics.
  * Bach et al. (2022) Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022.  PromptSource: An integrated development environment and repository for natural language prompts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 93–104, Dublin, Ireland. Association for Computational Linguistics.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations (ICLR 2015)_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org.
  * Borzunov et al. (2023) Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. 2023.  Petals: Collaborative inference and fine-tuning of large models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pages 558–568, Toronto, Canada. Association for Computational Linguistics.
  * Brachman et al. (2023) Michelle Brachman, Qian Pan, Hyo Jin Do, Casey Dugan, Arunima Chaudhary, James M. Johnson, Priyanshu Rai, Tathagata Chakraborti, Thomas Gschwind, Jim A Laredo, Christoph Miksovic, Paolo Scotton, Kartik Talamadupula, and Gegi Thomas. 2023.  Follow the successful herd: Towards explanations for improved use and mental models of natural language systems.  In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ , IUI ’23, page 220–239, New York, NY, USA. Association for Computing Machinery.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-SNLI: Natural language inference with natural language explanations.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Carneiro et al. (2021) Davide Carneiro, Patrícia Veloso, Miguel Guimarães, Joana Baptista, and Miguel Sousa. 2021.  A conversational interface for interacting with machine learning models.  In _XAILA @ ICAIL_.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2023) Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. 2023.  DISCO: Distilling counterfactuals with large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5514–5528, Toronto, Canada. Association for Computational Linguistics.
  * Dai et al. (2023) Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023.  AugGPT: Leveraging chatGPT for text data augmentation.  _arXiv_ , abs/2302.13007.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022.  8-bit optimizers via block-wise quantization.  In _International Conference on Learning Representations_.
  * Enguehard (2023) Joseph Enguehard. 2023.  Sequential integrated gradients: a simple but effective method for explaining language models.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 7555–7565, Toronto, Canada. Association for Computational Linguistics.
  * Feldhus et al. (2023) Nils Feldhus, Qianli Wang, Tatiana Anikina, Sahil Chopra, Cennet Oguz, and Sebastian Möller. 2023.  InterroLang: Exploring NLP models and datasets through dialogue-based explanations.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5399–5421, Singapore. Association for Computational Linguistics.
  * Ferreira and Monteiro (2020) Juliana J. Ferreira and Mateus S. Monteiro. 2020.  What are people doing about XAI user experience? a survey on AI explainability research and practice.  In _Design, User Experience, and Usability. Design for Contemporary Interactive Environments_ , pages 56–73, Cham. Springer International Publishing.
  * Frantar et al. (2023) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023.  OPTQ: Accurate quantization for generative pre-trained transformers.  In _The Eleventh International Conference on Learning Representations_.
  * Guo et al. (2022) Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022.  A survey on automated fact-checking.  _Transactions of the Association for Computational Linguistics_ , 10:178–206.
  * Jacovi et al. (2023) Alon Jacovi, Jasmijn Bastings, Sebastian Gehrmann, Yoav Goldberg, and Katja Filippova. 2023.  Diagnosing AI explanation methods with folk concepts of behavior.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23, page 247, New York, NY, USA. Association for Computing Machinery.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7B.  _arXiv_ , abs/2310.06825.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 22199–22213. Curran Associates, Inc.
  * Lakkaraju et al. (2022) Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chenhao Tan, and Sameer Singh. 2022.  Rethinking explainability as a dialogue: A practitioner’s perspective.  _HCAI @ NeurIPS 2022_.
  * Lee et al. (2023) Dong-Ho Lee, Akshen Kadakia, Brihi Joshi, Aaron Chan, Ziyi Liu, Kiran Narahari, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2023.  XMD: An end-to-end framework for interactive explanation-based debugging of NLP models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pages 264–273, Toronto, Canada. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Li et al. (2023a) Dawei Li, Yaxuan Li, Dheeraj Mekala, Shuyao Li, Yulin wang, Xueqi Wang, William Hogan, and Jingbo Shang. 2023a.  DAIL: Data augmentation for in-context learning via self-paraphrase.  _arXiv_ , abs/2311.03319.
  * Li et al. (2023b) Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. 2023b.  In-context learning with many demonstration examples.  _arXiv_ , abs/2302.04931.
  * Liang et al. (2023) Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2023.  MultiViz: Towards visualizing and understanding multimodal models.  In _The Eleventh International Conference on Learning Representations_.
  * Madsen et al. (2022) Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022.  Post-hoc interpretability for neural NLP: A survey.  _ACM Comput. Surv._ , 55(8).
  * Malandri et al. (2023) Lorenzo Malandri, Fabio Mercorio, Mezzanzanica Mario, and Nobani Navid. 2023.  ConvXAI: a system for multimodal interaction with any black-box explainer.  _Cognitive Computation_ , 15(2):613–644.
  * Marasovic et al. (2022) Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters. 2022.  Few-shot self-rationalization with natural language prompts.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 410–424, Seattle, United States. Association for Computational Linguistics.
  * Messina et al. (2022) Pablo Messina, Pablo Pino, Denis Parra, Alvaro Soto, Cecilia Besa, Sergio Uribe, Marcelo Andía, Cristian Tejos, Claudia Prieto, and Daniel Capurro. 2022.  A survey on deep learning and explainability for automatic report generation from medical images.  _ACM Comput. Surv._ , 54(10s).
  * Miller (1995) George A. Miller. 1995.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38(11):39–41.
  * Mosca et al. (2023) Edoardo Mosca, Daryna Dementieva, Tohid Ebrahim Ajdari, Maximilian Kummeth, Kirill Gringauz, and Georg Groh. 2023.  IFAN: An explainability-focused interaction framework for humans and NLP models.  _arXiv_ , abs/2303.03124.
  * Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023.  Orca: Progressive learning from complex explanation traces of GPT-4.  _arXiv_ , abs/2306.02707.
  * Ostermann et al. (2018) Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018.  SemEval-2018 task 11: Machine comprehension using commonsense knowledge.  In _Proceedings of the 12th International Workshop on Semantic Evaluation_ , pages 747–757, New Orleans, Louisiana. Association for Computational Linguistics.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  "why should i trust you?": Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 1135–1144, New York, NY, USA. Association for Computing Machinery.
  * Ross et al. (2021) Alexis Ross, Ana Marasović, and Matthew Peters. 2021.  Explaining NLP models via minimal contrastive editing (MiCE).  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3840–3852, Online. Association for Computational Linguistics.
  * Ross et al. (2022) Alexis Ross, Tongshuang Wu, Hao Peng, Matthew Peters, and Matt Gardner. 2022.  Tailor: Generating and perturbing text with semantic controls.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3194–3213, Dublin, Ireland. Association for Computational Linguistics.
  * Saakyan et al. (2021) Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021.  COVID-fact: Fact extraction and verification of real-world claims on COVID-19 pandemic.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2116–2129, Online. Association for Computational Linguistics.
  * Sarti et al. (2023) Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal. 2023.  Inseq: An interpretability toolkit for sequence generation models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pages 421–435, Toronto, Canada. Association for Computational Linguistics.
  * Shen et al. (2023) Hua Shen, Chieh-Yang Huang, Tongshuang Wu, and Ting-Hao Kenneth Huang. 2023.  ConvXAI: Delivering heterogeneous AI explanations via conversations to support human-AI scientific writing.  In _Computer Supported Cooperative Work and Social Computing_ , CSCW ’23 Companion, page 384–387, New York, NY, USA. Association for Computing Machinery.
  * Shin et al. (2021) Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021.  Constrained language models yield few-shot semantic parsers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7699–7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _Workshop at International Conference on Learning Representations_.
  * Slack et al. (2023) Dylan Slack, Satyapriya Krishna, Himabindu Lakkaraju, and Sameer Singh. 2023.  Explaining machine learning models with interactive natural language conversations using TalkToModel.  _Nature Machine Intelligence_.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3319–3328. PMLR.
  * Talmor et al. (2017) Alon Talmor, Mor Geva, and Jonathan Berant. 2017.  Evaluating semantic parsing against a simple web-based question answering model.  In _Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017)_ , pages 161–167, Vancouver, Canada. Association for Computational Linguistics.
  * Tenney et al. (2020) Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020.  The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 107–118, Online. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv_ , abs/2307.09288.
  * Wang et al. (2020) Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020.  Fairseq S2T: Fast speech-to-text modeling with fairseq.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 33–39, Suzhou, China. Association for Computational Linguistics.
  * Wang et al. (2023a) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2609–2634, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2023b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020.  Neural text generation with unlikelihood training.  In _International Conference on Learning Representations_.
  * Wiegreffe et al. (2022) Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022.  Reframing human-AI collaboration for generating free-text explanations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 632–658, Seattle, United States. Association for Computational Linguistics.
  * Wijekoon et al. (2023) Anjana Wijekoon, Nirmalie Wiratunga, Chamath Palihawadana, Ikeckukwu Nkisi-Orji, David Corsar, and Kyle Martin. 2023.  iSee: Intelligent sharing of explanation experience by users for users.  In _Companion Proceedings of the 28th International Conference on Intelligent User Interfaces_ , IUI ’23 Companion, page 79–82, New York, NY, USA. Association for Computing Machinery.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6707–6723, Online. Association for Computational Linguistics.
  * Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2023.  Large language models as optimizers.  _arXiv_ , abs/2309.03409.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
  * Zeiler and Fergus (2014) Matthew D. Zeiler and Rob Fergus. 2014.  Visualizing and understanding convolutional networks.  In _Computer Vision – ECCV 2014_ , pages 818–833, Cham. Springer International Publishing.
  * Zhang et al. (2023) Tong Zhang, X. Jessie Yang, and Boyang Li. 2023.  May i ask a follow-up question? understanding the benefits of conversations in neural network explainability.  _arXiv_ , abs/2309.13965.

##  Appendix A Explanation examples

User | LLMCheckup
---|---
|
Hi! What do you predict for instance with id 2435? |
10 most important features for it regarding attention? |
And starting it how would a new instance look like? |
Can you explain it in natural language? |
Table 3: Sample dialogues for welcome words, prediction (predict), feature attribution (nlpattribute), data augmentation (augment) and rationalization (rationalize) for the ECQA use case. User | LLMCheckup
---|---
How would you flip the prediction for it? |
Is it possible to retrieve an example that is similar to it? |
Table 4: Sample dialogues for counterfactual (nlpcfe), similar (similar) for
the ECQA use case.

Table 3 and Table 4 displays examples of explanations for each supported
explainability method. In each screenshot, the operation name is highlighted
in blue.

##  Appendix B Multi-prompt parsing

As a first step, MP queries the model about the main operation (e.g., feature
attribution, counterfactual generation, dataset statistics etc., see the full
list of the available operations in Appendix D). Next, depending on the chosen
operation, MP selects the operation-specific prompts with 2-7
demonstrations111111The number of demonstrations depends on the difficulty of
operation, e.g., how many attributes it may have. (user query and parsed
outputs examples) to generate the full parses that may include several
attributes. E.g., for the user input "What are the feature attributions for id
42 based on the integrated gradients?" we start by generating _nlpattribute_
and then augment the parse with the second prompt and transform it into
_filter id 42 and nlpattribute integrated_gradient_.

Since the output of the model is not constrained compared to GD, in the MP
setting, we need to check whether the model’s output matches any of the
available operations and if there is no exact match we employ SBERT and
compare the embedding similarity between the user input and the operations of
LLMCheckup (e.g., sometimes the model outputs synonyms but we need to match
the operation names exactly to ensure the correct parsing). Additionally, we
implement some checks to avoid possible hallucinations, e.g., if the model
outputs an id that is not present in the user input we remove it from the
parsed text.

Model | #max_new_tokens | Accuracy
---|---|---
Falcon | 10 | 64.15
Falcon | 20 | 64.15
Pythia | 10 | 75.91
Pythia | 20 | 63.03
Llama2 | 10 | 74.79
Llama2 | 20 | 82.35
Llama2-GPTQ | 10 | 82.63
Llama2-GPTQ | 20 | 87.40
Mistral | 10 | 84.87
Mistral | 20 | 71.43
Mistral-GPTQ | 10 | 78.71
Mistral-GPTQ | 20 | 68.91
Stable Beluga 2 | 10 | 88.24
Stable Beluga 2 | 20 | 86.55

Table 5:  Parsing accuracy (in %) using MP with different number of maximum
new tokens. Note that for the Llama2-7b and Mistral-7b models, we offer
various options for quantization. In this case, we have chosen GPTQ as the
representative method.

Table 5 summarizes our parsing evaluation results for different models with
different number of ’max_new_tokens‘ for generation specified in the
configuration file. We found that this parameter is very important for the
performance of the MP parser. Pythia-2.8B, Mistral-7B and Stable Beluga 2
achieve better scores with less tokens to generate (10) and Llama2-7B shows
better performance with larger number of tokens (20). The only model that is
unaffected by the ’max_new_tokens‘ parameter is Falcon-1B which also has the
smallest size. The majority of the models perform better when allowed to
generate only brief texts and if they are given more tokens to generate they
tend to hallucinate and detract from the actual parsing task.

Originally, we treated all models in the same way during parsing, however,
after looking at some generated outputs we realized that Falcon-1B and
Pythia-2.8B are not good at extracting ids and often can only recognize the
main operation. Hence, for these models we have an additional step that
extracts a potential id from the user input and adds it to the parsed
operation.

As expected, larger models tend to perform better than the ones with less
parameters. However, we also found that the quantized Llama model outperforms
its full (non-quantized) version on the parsing task.

##  Appendix C Follow-up suggestions

We group the suggestions into several categories that include metadata
operations (e.g., dataset statistics, model types etc.), prediction-related
operations (e.g., predict, count or show mistakes), explanation-based
operations (e.g., attributions for top $k$, attention scores and integrated
gradients or free-text rationale), NLU (similarity and keywords) and input
perturbations (counterfactuals and data augmentation).

The user always has an option to decline the suggestion and ask something
different. We check whether the user agrees to the LLMCheckup suggestions by
computing the similarity scores between the input and the confirm/disconfirm
templates with SBERT.

##  Appendix D Supported operations

We define an operation as a mapping between (a part of) an SQL-type query and
a function executed by LLMCheckup. Table 6 enumerates all operations
facilitated by LLMCheckup, including all explainability methods mentioned in
§3 and other supplementary operations.

Filt. | filter(id) | Access single instance by its ID
---|---|---
includes(token) | Filter instances by token occurrence
Prediction | predict(instance)∗ | Get the prediction of the given instance
randompredict(number) | Precompute a subset of instances at random
mistakes(subset) | Count number of wrongly predicted instances
score(subset, metric) | Determine the relation between predictions and labels
Data | show(list) | Showcase a list of instance
countdata(list) | Count number of instances within the given list
label(dataset) | Describe the label distribution across the dataset
Meta | data() | Information related to the dataset
model() | Metadata of the model
About | function() | Inform the functionality of the system
self() | Self-introduction
qatutorial(op_name) | Provide explanation for the given operation
Logic | and(op1, op2) | Concatenation of multiple operations
or(op1, op2) | Selection of multiple filters
Explanation | nlpattribute(inst., topk, method_name)∗ | Provide feature attribution scores
raionalize(instance)∗ | Explain the output in natural language
cfe(instance)∗ | Generate counterfactuals
similarity(instance, number)∗ | Output top $k$ similar instances in the dataset
augment(id)∗ | Augment the input text

Table 6:  Operations marked with (∗) support the use of custom inputs.

##  Appendix E Details on feature attribution

In LLMCheckup, we do not exhaustively employ all Inseq’s methods for feature
attribution. Instead, we selectively choose certain representative methods
from our perspective. Nevertheless, we would like to emphasize that it is
straightforward to incorporate addition methods such as Saliency Simonyan et
al. (2014), Occlusion Zeiler and Fergus (2014), Sequential Integrated
Gradients Enguehard (2023).

##  Appendix F Interface

Figure 3 display the user interface of LLMCheckup. Within the chat window, a
user’s question requesting the explanation for instance with ID 4180 in
natural language is shown, followed by a response that includes the
corresponding rationale. On the right side of the window, there is a prompt
modification section where the Optimization by Prompting (OPRO) strategy is
chosen for rationalization. The icons associated with each strategy describe
them in detail when they are hovered over, including the corresponding prompts
that can be appended after the system prompt we have designed.

Figure 3: LLMCheckup interface with initial welcome message, generated free-
text rationale and sample generator buttons. Expert level of knowledge in XAI
is selected and Optimization by Prompting (OPRO) strategy is used. For an
example dialogue, see Table 3 and Table 4.

##  Appendix G QA tutorial

Figure 4 shows tutorials for data augmentation with different level of
expertise in XAI.

(a) QA Tutorial for data augmentation with beginner level of knowledge in XAI.

(b) QA Tutorial for data augmentation with expertise level of knowledge in
XAI.

(c) QA Tutorial for data augmentation with expert level of knowledge in XAI.

Figure 4: QA tutorial with different knowledge level in XAI.

##  Appendix H External information retrieval

Figure 5 shows the external information retrieval for an instance from COVID-
Fact.

Figure 5: External information retrieval of an instance from COVID-Fact.