URL

    Uniform Resource Locator
AI

    Artificial Intelligence
ML

    Machine Learning
DS

    Data Science
GNN

    Graph Neural Network
HCI

    Human-Computer Interaction
CSED

    Computer Science Education
CSCW

    Computer-Supported Cooperative Work
UI

    User Interface
GUI

    Graphical User Interface
AIOP

    AI Operator
AST

    Abstract Syntax Tree
NLP

    Natural Language Processing
NMT

    Neural Machine Translation
API

    Application Programming Interface
HTTP

    Hypertext Transfer Protocol

#  Documentation Matters: Human-Centered AI System to Assist Data Science Code
Documentation in Computational Notebooks

April Yi Wang<EMAIL_ADDRESS>University of MichiganUSA ,  Dakuo Wang
<EMAIL_ADDRESS>IBM ResearchUSA ,  Jaimie Drozdal<EMAIL_ADDRESS>Rensselaer Polytechnic InstituteUSA ,  Michael Muller
<EMAIL_ADDRESS>IBM ResearchUSA ,  Soya Park<EMAIL_ADDRESS>MIT
CSAILUSA ,  Justin D. Weisz<EMAIL_ADDRESS>IBM ResearchUSA ,  Xuye Liu
<EMAIL_ADDRESS>Rensselaer Polytechnic InstituteUSA ,  Lingfei Wu
<EMAIL_ADDRESS>IBM ResearchUSA and  Casey Dugan<EMAIL_ADDRESS>IBM
ResearchUSA

###### Abstract.

Computational notebooks allow data scientists to express their ideas through a
combination of code and documentation. However, data scientists often pay
attention only to the code, and neglect creating or updating their
documentation during quick iterations. Inspired by human documentation
practices learned from 80 highly-voted Kaggle notebooks, we design and
implement Themisto, an automated documentation generation system to explore
how human-centered AI systems can support human data scientists in the machine
learning code documentation scenario. Themisto facilitates the creation of
documentation via three approaches: a deep-learning-based approach to generate
documentation for source code, a query-based approach to retrieve online API
documentation for source code, and a user prompt approach to nudge users to
write documentation. We evaluated Themisto in a within-subjects experiment
with 24 data science practitioners, and found that automated documentation
generation techniques reduced the time for writing documentation, reminded
participants to document code they would have ignored, and improved
participants’ satisfaction with their computational notebook.

code summarization, computational notebooks, code documentation

††copyright: acmcopyright††journal: TOCHI††journalyear: 2022††journalvolume:
29††journalnumber: 2††article: 17††publicationmonth: 1††price: 15.00††doi:
10.1145/3489465††ccs: Human-centered computing Interactive systems and
tools††ccs: Human-centered computing Empirical studies in HCI††ccs: Computing
methodologies Natural language generation††ccs: Software and its engineering
Documentation

##  1\. introduction

Documenting the story behind code and results is critical for data scientists
to collabrate effectively with others, as well as  their future selves (Kery
and Myers, 2017; Piorkowski et al., 2021; Kross and Guo, 2021; Muller et al.,
2019). The story, code, and computational results together construct a
computational narrative. Unfortunately, data scientists often write messy and
drafty analysis code in computational notebooks as they need to quickly test
hypotheses and experiment with alternatives. It is a tedious process for data
scientists to then manually document and refactor the raw notebook into a more
readable computational narrative, thus many people neglect to do so (Rule et
al., 2018b).

Many efforts have sought to address the tension between _exploration_ and
_explanation_ in computational notebooks. For example, researchers have
explored the use of code gathering techniques to help data scientists organize
cluttered and inconsistent notebooks (Head et al., 2019), as well as
algorithmic and visualization approaches to help data scientists forage past
analysis choices (Kery et al., 2019). But these efforts focus on the cleaning
and organizing of existing notebook content, instead of creating the new
content. Another work developed a chat feature that enables data scientists to
have simultaneous discussions while coding in a notebook (Wang et al., 2020b),
and linked their chat messages as documentations to relevant notebook elements
as in Google Docs (Wang, 2016). However, these chat messages are too
fragmented and colloquial to be used for documentation; besides, in real
practice  data scientists and business analysts rarely work on notebooks at
the same time and actively message each other.

We began our project by asking, “What makes a well-documented notebook?” To
answer this question, we first conducted an in-depth analysis of how human
data scientists document notebooks. Publicly shared user notebooks on Githubs
are often not well documented (Rule et al., 2018b), thus we look up to a
special set of notebooks – the highly-voted notebooks users submitted to
Kaggle competitions. We conducted a formative study with a sample of 80 of
these notebooks, and our interative indepth coding analysis suggested these 80
notebooks have much better documentations in comparison to the corpus reported
in previous literature (Rule et al., 2018b). Thus, we refer to them as “well-
documented” notebooks.

Our coding process of these 80 notebooks also revealed a taxtonomy of nine
categories (e.g., Reason, Process, Result) for the documentation content,
which reflects the thought processes and decisions made by the notebook owner.
These findings together with the insights from related work motivate us to
consider AI automation as a potential solution to support the human process of
crafting documentation.

We propose Themisto, an automated code documentation generation system that
integrates into the Jupyter Notebook environment. To support the diverse types
of documentation content and to complement the AI limitations, Themisto
incorporate three distinct approaches: a deep-learning-based approach to
automatically generate new documentation for source code (fully automated); a
query-based approach to retrieve existing documentation from online
Application Programming Interface (API) websites for third party packages and
libraries (fully automated); and a prompt-based approach to give users a start
of the sentence and encourage them to complete the sentence that serves as
documentation (semi automated).

We evaluated Themisto in a within-subjects experiment with 24 data science
practitioners. We found that Themisto reduced the time for data scientists to
create documentation, reminded them to document code they would have ignored,
and improved their satisfaction with their computational notebooks. Meanwhile,
the quality of the documentation produced with Themisto are about the same as
what data scientists produced on their own.  Base on these findings, we re-
imagine that the code documentation task can be conducted in a Human-AI
Collaboration fasion in the future, where this joint effort may have unique
advantages in comparison to the solo effort of a human alone.

Our paper provides a three-fold contribution to the HCI and data science
practitioner communities:

  * •

providing an empirical understanding of best practices of how human
documenting a notebook through an analysis of highly-rated Kaggle notebooks,

  * •

demonstrating the design of  a human-centered AI system that can collaborate
with human data scientists to create high-quality computational narratives,

  * •

reporting empirical evidence that Themisto can collaborate with data
scientists to generate high quality and highly-satisfied computational
notebooks in much less time.

##  2\. Related Work

Our work builds on top of both Human-Computer Interaction (HCI) and Machine
Learning (ML) fields. Thus, our literature review briefly summarizes the work
of both, with a focus on the following three topics: computational notebook
management, code documentation supporting systems, and neural-network-based
code summarization.

Figure 1. Computational notebooks allow data scientists to create (A) markdown
cells and (B) code cells, and view (C) code output in the same environment.
Together, the variety of media including text explanations, graphs, forms,
interactive visualizations, code segments and their outputs — weaves into
computational narratives.

###  2.1. Computational Notebooks as Computational Narrative

Computational notebooks allow data scientists to weave together a variety of
media, including text explanations, graphs, forms, interactive visualizations,
code segments, and their outputs, into computational narratives (as shown in
Figure 1). These computational narratives enable literate programming (Knuth,
1984) and allow data scientists to effectively create, communicate, and
collaborate on their analysis work. The data scientist community has widely
adopted notebook systems (e.g., Jupyter Notebook (Jupyter, 2015) and Jupyter
Lab (Jupyter, [n.d.])) as their main working envrioment (Perkel, 2018).

However, it is not easy for data scientists to create a computational
narrative while they are coding for rapid exploration. Data scientists often
need to explore diverse sets of hypotheses and theories (Liu et al., 2019;
Rehman, 2019). Active exploration of alternatives increases the workload for
data scientists to track the history of their experimentation (Kery et al.,
2018). Thus, documenting for those alternatives will pose more workload to
data scientists, sometimes interference with their cognitive process of
coding, and is hardly rewarding because many those alternatives will be
discarded in later versions.

Because creating and maintaining a clean computational narrative is often an
expensive and tedious process, many computational notebooks shared within open
communities are not appropriately documented. For example, Rule et al.
examined 1 million open-source computational notebooks from Github and found
that one in four lacked any sort of written documentation (Rule et al.,
2018b). In addition, they analyzed a sample of 221 academic computational
notebooks, which they considered are higher quality notebooks, and found that
academic computational notebooks contained text cells for introduction,
describing analytical steps, explaining the reasoning, and discussing results.

Poor documentation hinders the readability and reusability of the notebooks
that are shared with other collaborators or even with one’s future self
(Chattopadhyay et al., 2020). Recently, various groups of researchers have
developed a wide range of tools to help data scientists to manage their
“messy” computational notebooks.  Notably, Lau et al. summarized the design
space of computational notebooks which covered an overview for improving
explanations in computational notebooks (Lau et al., 2020). Many strategies
interact with markdown comments. For example, facilitating cell folding could
help surface important markdown cells (Rule et al., 2018a); Kery et al.
designed Verdant (Kery et al., 2019), a lightweight local versioning plugin
for Jupyter Lab, that uses algorithmic and visualization techniques for data
science workers to better forage their past analysis choices; Woods et al
argued (Wood et al., 2018) for simpler and richer narratives; Head et al. used
code gathering tools to help data scientists trace back to the computational
code from an end result (Head et al., 2019); Wenskovitch et al. designed an
interactive tool that produced a visual summary of the structure of a
computational notebook (Wenskovitch et al., 2019); Wang et al. proposed
capturing the contextual connections between notebook content and discussion
messages to help data science teams reflect on their decision making process
(Wang et al., 2020b).

However, despite the wide variety of approaches to helping data scientsts
manage their notebooks, none of these tools directly aids data scientists in
creating new, rich, descriptive contents to document their computational
notebooks, and to improve the quality of the computational narrative. Recent
research works have proposed to use AI solutions to automate the various tasks
along a data science project, such as the model training, model selection, and
feature selection, and these technology are commonly refered as AutoML (Wang
et al., 2021a; Liu et al., 2020). The research gap and the AutoML techniques
motivate us to design and build an AI system to support data scientists to
better document their code and to produce higher qualitive computational
narratives.

But what makes up a good computational narrative? Despite the portrait of not-
so-good notebooks on Github (Rule et al., 2018b), we need further
understanding and role models for well-documented computational narratives.
Thus, we decided to first conduct an in-depth analysis of some highly-voted
notebooks on Kaggle competetion111https://www.kaggle.com/. Kaggle competition
provides a platform where organizations post datasets as challenges, and many
data scientists submit their notebooks as solutions to a challenge. If a
solution has the highest accuracy, it wins the competition. But those winning
solutions are often not the most voted ones, as community members voted on
readability and completeness of the computational narrative.

###  2.2. Code Documentation in Software Engineering

Documentation plays an important role in software programming. Programmers
write comments in their source code to make the code easier for both
themselves and others to understand (Padioleau et al., 2009). Writing clear
and comprehensive documentation is critical to software development and
maintenance (de Souza et al., 2005; Kajko-Mattsson, 2005; Shi et al., 2011;
Roehm et al., 2012; Maalej and Robillard, 2013). However, writing
documentation itself is a time-consuming task. And that is why documentation
practices in open source communities are widely perceived to be of low
quality, due in part to low levels of intrinsic enjoyment for doing
documentation work (Geiger et al., 2018).

To save time in creating documentation, template-based approaches are often
used to help developers annotate their source code. For example, tools like
JavaDoc222JavaDoc:
https://docs.oracle.com/javase/8/docs/technotes/tools/windows/javadoc.html and
JSDoc333JSDoc: https://devdocs.io/jsdoc/ allow programmers to annotate their
code with tags (e.g., @param, @return) and then automatically generate
documentation using these tags. This approach helps programmers create
documentation for others and works especially well for documenting APIs, where
method signatures and variable types are important pieces of information and
can easily be documented from tags. Although, such methods may not work in the
rapid, experimental nature of data science work, because data scientists may
be particularly reluctant to create and maintain high-quality documentation of
their work. Furthermore, these methods can not capture other aspects of
documentation important in data science, such as how a data set was
constructed, the intent behind an analysis, or a description of why an
experiment was successful or not.

Recently, some researchers have put forth proposals for better documenting the
specific artifacts invovled in a data science workflow, i.e., the data set and
the machine learning model (Gebru et al., 2018; Holland et al., 2018; Mitchell
et al., 2019; Arnold et al., 2019). Notably, Gebru et al. (Gebru et al., 2018)
and Holland et al. (Holland et al., 2018) proposed both qualitative and
quantitative  guidelines for documenting a dataset, so that the dataset
creators and maintainers can follow these guidlines to document useful
information for the data. Similarly, Mitchell et al. (Mitchell et al., 2019)
and Arnold et al. (Arnold et al., 2019) explored using formulas to document
the machine learning model artifacts, and sharing such formulas with others.
These approaches are inline with what is called provenance, which refers to
tracking what has been done with code and data over time – typically to aid
reproducibility of results – using applications such as noWorkflow and
YesWorkflow (Pimentel et al., 2016), ReproduceMeGit (Samuel and König-Ries,
2020), and Provbook (Samuel and König-Ries, 2018). However, this approach
focus more on the the dataset and the model artifact in the final product of a
data science project, and supporting data scientists to create a “factsheet”
for these artifacts for the non-technical consumers. Instead, we want to
support data scientists to better create the documentation during the process
of creating models and data science products, and such documentation, together
with the code as a computational narrative, is primarily for other technical
users to understand and to reuse.

In addition to the various ways of generating new documentation for code,
there is another research line that focuses on improving the usability of
documentation, as novice programmers may find it difficult to read and use API
documentation (Horvath et al., 2019). Oney et al. proposed linking interactive
documentation and example code in an editor to help novice programmers better
understand the external documentation and write code (Oney and Brandt, 2012).
We believe this approach of linking code with external documentation is a
promising way to help data scientists to create more usable documentations,
and we will also implement this retrieval-based approach in our system.

###  2.3. GNN-Based Automatic Code Summarization System

Automatic code summarization is a rapidly-expanding research topic in the
Natural Language Processing (NLP) and ML communities (Eddy et al., 2013; Hu et
al., 2018; Alon et al., 2018; Iyer et al., 2016a; LeClair et al., 2020). The
automatic code summarization task can be considered as a translation task,
which takes a code snippet as the input sequence, and translating it into a
natural language description of the code as an output sequence.

Early work primarily used predefined templates and heuristics to produce code
summaries (e.g., (Eddy et al., 2013; Sridhara et al., 2010)). Recent studies
have taken the advantage of modern deep neural network architectures to
generate the summary for the code (e.g. (Hu et al., 2018; Alon et al., 2018;
Iyer et al., 2016b)). Motivated by the language translation task (e.g.,
English to French), most of these learning-based approaches are based on the
Neural Machine Translation (NMT) model architecture (Luong et al., 2015). This
architecture breaks code into a sequence of input tokens and produce the
summarization text as a sequence of output tokens.

However, this sequence-to-sequence approach does not work well in practice
because source code is not just a stream of tokens. There is additional
semantic information that is lost when processing source code in this way.
LeClair et al. proposed improving code summarization through the use of Graph
Neural Networks (LeClair et al., 2020). The GNN model can take in both the
code sequence and its Abstract Syntax Tree (AST) structure (refer to Fig 5 for
an example) as input to generate summary sentences as output. Their approach
achieved better accuracy than the baseline algorithms.

Our work explores neural-network-based automatic code summarization techniques
to support document writing in computational notebooks. To our acknowledge,
there has been little discussion in the HCI community on leveraging automatic
code summarization techniques to improve documentation. Furthermore, we
suspect that the automation approach alone may not work well in the
documentation creation task, as data science is a highly interdisciplinary
field that requires various human expertises to explain and interpret.
Inspired by prior studies that implement Artificial Intelligence (AI) systems
to work together with human (Louie et al., 2020; Wang et al., 2020a), we
believe the system will work better if it has both the automated documentaiton
capability and the capability that allows users to directly manipulate the
documentation. However, what types of documentation may be better suited for
AI to do, and what works should the system leave to human data scientists?
This is a design question that requires further exploration of the best
practices for creating notebooks. Thus, we start this project with a formative
study to fill this research gap.

##  3\. Formative Study

In order to build a useful system that can support data scientists to create
documentations and to improve their computational narrative’s quality, we
first need to explore and understand the characteristics of good
documentations in high-quality notebooks. What does a well-documented
computational narrative look like? We identify well-documented computational
narratives with ratings from a broader data scientist community (Kaggle), and
analyze their characteristics specifically around the documentation. We
consider the community voting number is a good indicator to reflect a
computational notebook’s quality for our research goal. Based on this premise,
we then conduct a formative study to analyze the characteristics of a set of
most voted computational narratives, and explore how the data scientists
create documentations for these notebooks.

###  3.1. Data Collection

We collected notebooks from two popular Kaggle competitions — House Price
Prediction444https://www.kaggle.com/c/house-prices-advanced-regression-
techniques and Titanic Survival
Prediction555https://www.kaggle.com/c/titanic/. We chose these two
competitions because they are the most popular competitions (5280 notebooks
submitted for House Price and 6300 notebooks submitted for Titanic Survival)
and because many data science courses use these two competitions as a tutorial
for beginners (Bai and Hu, 2018; Fernandez-Bes et al., 2016).

We collected the top 1% of the submitted notebooks from each competition based
on their voting numbers, which resulted in 53 for House Price and 63 for
Titanic Survival. We then filtered out the notebooks that were not written in
English and the ones that are not relevant to the particular challenge (e.g.,
a computational notebook as a tutorial on how to save memories can win lots of
votes, but it is not a solution to the challenge), which returned 80 valid
notebooks for analysis (39 for House Price and 41 for Titanic Survival).

Figure 2.  We replicated the notebook-level descriptive analysis by Rule et
al. (Rule et al., 2018b) to the 80 well-documented notebooks on Kaggle. The
left side represents the descriptive visualization of the 80 well-documented
computational notebooks from Kaggle (noted as Sample A) and the right side
represents the descriptive visualization of the 1 million computational
notebooks on Github (noted as Sample B). The highly-voted notebooks on Kaggle
are better documented compared to the Github notebooks.

###  3.2. Data Analysis

Five members of the research team conducted an iterative open coding  process
to analyze the collected notebooks. Differing from (Rule et al., 2018b), where
their qualitative coding  stopped at the notebook level, our analysis goes
deep to the cell granularity: we code each cell’s purposes and types of
content; and which step (stages) in the data science lifecycle that the cell
belongs (e.g., data cleaning or modeling training (Zhang et al., 2020; Wang et
al., 2021b)). Our analysis covered 4427 code cells and 3606 markdown cells
within the 80 notebooks. Each notebook took around 1 hour to code as we coded
the notebook at the cell level.

Each coder independently analyzed the same six notebooks to develop a
codebook. After discussing and refining the codebook, they again went back to
recode the six notebooks and achieved pair-wise inter-rater reliability ranged
0.78-0.95 (Cohen’s $\kappa$). After this step, the five coders divided and
coded the remaining notebooks.

Table 1. We identified 9 categories based on the purpose of markdown cells. Note that a markdown cell may belong to multiple categories of contents or none of the categories. Category |  N |  Description |  Example
---|---|---|---
Process |  2115
(58.65%) |  The markdown cell describes what the following code cell is doing. This always appears before the relevant code cell. |  `Transforming Feature X}\newline \mintinline`html to a new binary variable
Headline |  1167
(32.36%) |  The markdown cell contains a headline in markdown syntax. The cell is used for navigation purposes or marking the structure of the notebook. It may be relevant to a nearby code cell. |

    # Blending Models}\\
            \midrule
            Result & 692 \newline (19.19\%) & The markdown cell explains the output. This type always appears after the relevant code cell. & \mintinline

html It turns out there is a
`long tail of outlying}\newline \mintinline`html properties…
`1. Apply models}\newline\mintinline`html2. Get cross validation
`scores} \newline \mintinline`html3. Calculate the mean
`Gradient Boosting} \newline\mintinline`html Regression Refer

    [here](https://...)}\\
            \midrule
            Meta-Information & 141 \newline (3.91\%) & The markdown cell contains meta-information such as project overview, author’s information, and a link to the data sources. This often is not relevant to a specific code. & \mintinline

html The purpose of this
`notebook is to build a}\newline \mintinline`htmlmodel with Tensorflow.
Summary |  51
(1.41%) |  The markdown cell summarizes what has been done so far for a section or a series of steps. This often is not relevant to a specific code. |  `**In summary**}\newline\mintinline`html By EDA we found a strong
`impact of features like}\newline\mintinline`html Age, Embarked..

###  3.3. Results

We found that these 80 well-documented computational notebooks all contain
rich documentation. In total, we identified nine categories for the content of
the markdown cells. In addition, we found the markdown cells covered four
stages and 13 tasks of the data science workflow (Wang et al., 2021b). Note
that a markdown cell may belong to multiple categories.

####  3.3.1. Descriptive statistics of the notebook.

We found that on average, each notebook contains 55.3 code cells and 45.1
markdown cells. We replicated  the notebook descriptive analysis that Rule et
al. used to analyze 1 milion computational notebooks on Github (Rule et al.,
2018b). As shown in Figure 2, the left side represents the descriptive
visualization of the 80 well-documented computational notebooks from Kaggle
(noted as Sample A) and the right side reprersents the descriptive
visualization of the 1 milion computational notebooks on Github (noted as
Sample B). We found that the Sample A has more total cells per notebook
(Median = 95) than Sample B (Median = 18). Sample A has roughly equal ratio of
markdown cells and code cells per notebook, while Sample B is unbalanced with
majority cells being code cells. Notably, Sample A has more total words in
markdown cells (Median = 1728) than Sample B (Median = 218). This result
indicates that the 80 well-documented computational notebooks are better
documented than general Github notebooks.

####  3.3.2. Data scientists use markdown cells to document a broad range of
topics.

As shown in Table 1, our analysis revealed that markdown cells are mostly used
to describe what the adjacent code cell is doing (Process, 58.65%). Second to
the Process category, 32.36% markdown cells are used to specify a headline for
organizing the notebook into separate functional sections and for navigation
purposes (Headline).

Markdown cells can also be used to explain beyond the adjacent code cells. We
found that many markdown cells are created to describe the outputs from code
execution (Result, 19.19%), to explain results or critical decisions (Reason,
6.30%), or to provide an outline for the readers to know what they are going
to do in a list of todo actions (Todo, 5.60%), and/or to recap what has been
done so far (Summary, 1.41%).

We observed that 11.48% markdown cells explain what a general data science
concept means, or how a function works ( Background Knowledge), while 5.54%
markdown cells are connected with external references for readers to further
explore the topics (Reference). We believe these are the extra efforts that
the notebook owners dedicated, to attract a broader audience, especially
beginners in the Kaggle community. In addition, we found that authors
approached the story in different styles. For example, some authors want to
leave their own signature, and so they spend spaces at the beginning of the
notebooks to debrief the project, to add the author’s information, or even to
add their mottos (Meta-Information, 3.91%). Some authors prefer to use concise
and accurate language to convey important information; while others write
documentation in more creative and entertaining ways — for example, making
analogies between data science workflow and starting a business.

Table 2. We coded each markdown cell to which data science stage (or task) they belong. We identified 4 stages with 13 tasks out of the data science lifecycle (Wang et al., 2021b). Note that a markdown cell may belong to multiple stages or none of the stages. Stage | Total |  Task | N
---|---|---|---
Environment Configuration | 162 (4.49%) |  Library Loading | 33 (0.92%)
Data Loading | 129 (3.58%)
Data Preparation and Exploration | 1336 (37.05%) |  Data Preparation | 91 (2.52%)
Exploratory Data Analysis | 960 (26.62%)
Data Cleaning | 285 (7.90%)
Feature Engineering and Selection | 375 (10.40%) |  Feature Engineering | 120 (3.32%)
Feature Transformation | 178 (4.94%)
Feature Selection | 77 (2.14%)
Model Building and Selection | 994 (27.57%) |  Model Building | 247 (6.85%)
Data Sub-Sampling and Train-Test Splitting | 61 (1.69%)
Model Training | 377 (10.45%)
Model Parameter Tuning | 81 (2.25%)
Model Validation and Assembling | 288 (6.32%)

####  3.3.3. Data science stages.

We coded markdown cells based on where they belong in the data science
workflow (Wang et al., 2019).  As shown in Table 2, we identified four stages
and 13 tasks. The four stages include environment configuration (4.50%), data
preparation and exploration (37.05%), feature engineering and selection
(10.40%), and model building and selection (27.57%). At the finer-grained task
level, in particular, notebook authors create more markdown cells for
documenting exploratory data analysis tasks (26.62%) and model training tasks
(10.45%). The rest of the markdown cells are evenly distributed along with
other tasks.

###  3.4. Design Implications

In summary, our analysis of markdown cells in well-documented notebooks
suggests that data scientists document various types of content in a notebook,
and the distribution of these markdown cells generally follows an order of the
data science lifecycle, starting with data cleaning, and ending with model
building and selection. Based on these findings, we synthesize the following
actionable design considerations:

  * •

The system should support more than one type of documentation generation. Data
scientists benefit from documenting not only the behavior of the code, but
also interpreting the output, and explaining rationales. Thus, a good system
should be flexible to support more than one type of documentation generation.

  * •

Some types of documentations are highly related to the adjacent code cell. We
found at least the Process, Result, Reason, and Reference types of
documentations are highly related to the adjacent code cell. To automatically
generate interpretations of results or rationale for a decision may be hard,
as both involve deep human expertise. But, with the latest neural network
algorithms, we believe we can build an automation system to generate Process
type of documentation, and we can also retrieve Reference for a given code
cell.

  * •

There are certain types of documentations that are irrelevant to the code.
Various types of documentations do not have a relevant code piece upon which
the automation algorithm can be trained. Together with the Reason and Result
types, the system should also provide a function that the human user can
easily switch to the manual creation mode for these types.

  * •

For different types of documentation, it could be at the top or the bottom of
the related code cell. This design insight is particularly important to the
Process, Result, and Reason types of documentation. It may be less preferable
to put Result documentation before the code cell, where the result is yet to
be rendered. The system should be flexible to render documentation at
different relative locations to the code cell.

  * •

External resources such as Uniform Resource Locators and the official API
descriptions may also be useful. Some types of documentation, such as
Background Knowledge and Reference, are not easy to be generated with the NN-
based models, but they are easy to retrieve from the Internet. So the system
should incorporate the capability to fetch relevant web content as candidate
documentation.

  * •

There is an ordinality in markdown cells that is aligned with the data science
project’s lifecycle. The system should consider that Library Loading types of
cells are often at the beginning section of the notebook, and the Model
Training type of content may be more likely to appear near the end of the
notebook. In our system prototype, though, we did not take this design
consideration into account, it will be our future work.

  * •

The notebook would be nice to have documentation with a problem overview at
the beginning and a summary at the end. We considered this design implication
not in the system design, but our evaluation study design. For the two
barebone notebooks we used in the experiment, we always provide a problem
overview as a markdown cell at the top of the notebook.

##  4\. The Themisto System: Design and Implementation

Based on findings from the formative study and design insights from related
works, we design and implement Themisto, an automatic documentation generation
system that supports data scientists to write better-documented computational
narratives. In this section, we present the system architecture, the user
interface design, and the core technical capability of generating
documentation.

###  4.1. System Architecture

The Themisto system has two components: the client-side User Interface (UI) is
implemented as a Jupyter Notebook plugin using TypeScript code, and the
server-side backend is implemented as a server using Python and Flask.

The client-side program is responsible to render user interface, and also to
monitor the user actions on the notebook to edits in code cells. When the
user’s cursor is focused on a code cell, the UI will send the current code
cell content to the server-side program through Hypertext Transfer Protocol
(HTTP) requests.

The server-side program takes the code content and generates documentation
using both the deep-learning-based approach and the query-based approach. For
the deep-learning-based approach, the server-side program first tokenizes the
code content and generates the AST. It then generates the prediction with the
pre-trained model. For the query-based approach, the server-side program
matches the curated API calls with the code snippets and returns the pre-
collected descriptions. For the prompt-based approach, the server-side program
sends different prompts (e.g., for interpreting result or for explaining
reason) base on the output type of the code cell.

###  4.2. User Interface Design

Figure 3. The Themisto user interface is implemented as a Jupyter Notebook
plugin: (A) When the recommended documentation is ready, a lightbulb icon
shows up to the left of the currently-focused code cell. (B – D) shows the
three options in the dropdown menu generated by Themisto, (B) A documentation
candidate generated for the code with a deep-learning model, (C) A
documentation candidate retrieved from the online API documentation for the
source code, and (D) A prompt message that nudges users to write documentation
on a given topic.

Figure 3 shows the user interface of Themisto as a Jupyter Notebook plugin.
Each time the user changes their focus on a code cell, as they may be
inspecting or working on the cell, the plugin is triggered. The plugin sends
the user-focused code cell’s content to the backend. Using this content, the
backend generates a code summarization using the model and retrieves a piece
of documentation from the API webpage. When such a documentation generation
process is done, the generated documentation is sent from the server-side to
the frontend, and a light bulb icon appears next to the code cell, indicating
that the there are recommended markdown cells for the selected code cell (as
shown in Figure 3.A).

When a user clicks on the light bulb icon  which appears next to any selected
code cells, Themisto render all the three options in the dropdown menu: (1) a
deep-learning-based approach to generate documentation for source code (Figure
3.B); (2) a query-based approach to retrieve the online API documentation for
source code (Figure 3.C); and (3) a user prompt approach to nudge users to
write more documentation (Figure 3.D). If the user likes one of these three
candidates, they can simply click on one of them, and the selected
documentation candidate will be inserted into above the code cell (if it is
the Process, Reference, or Reason type), or below it (if it is the Result
type).

Figure 4. An illustration of the three different approaches for documentation
generation in Themisto.

###  4.3. Three Approaches for Documentation Generation

In this subsection, we describe the rationale and implementation detail of the
three different approaches for documentation generation (Figure 4):

  * •

Our formative study suggests that the system should be able to generate
multiple types of documentation (e.g., Process, Result,  Background Knowledge,
Reason, and Reference).

  * •

Some types of documentation can be directly derived from the code, thus the
automated approaches can help. The Process type of documentation directly
describes the coding process, and existing ML literature suggest that the
deep-learning-based approach is most suitable for generating it; The Reference
type does not need a learning-based approach, it can be achieved with a
traditional query-based approach, which locates and retrieves the most
relevant online documentation as candidates;

  * •

Some others types of documentations (e.g., Education, Result, and Reason) are
not directly related to the code, thus the fully automated approaches are not
capable of generating such contents. We design the prompt-based approach for
users to complete the generation process.

####  4.3.1. Deep-Learning-Based Approach

We trained a  deep learning model666 We release a larger dataset and a refined
version of the model in a separate paper (Liu et al., 2021). using the Graph-
Neural-Network architecture based on LeClair et al. (LeClair et al., 2020).
These GNN models can take both the source code’s structure (extracted as AST)
and the source code’s content as input. Thus, it outperforms the traditional
sequence-to-sequence model architectures, which only takes the source code’s
content as an input sequence, in source code summarization tasks for Python
code777All the collected data science notebooks are in Python.. We did not
consider T5, BerT or GPT-3 architectures as these models can take minutes to
make one inference (i.e., generate one summary) even with a cluster of GPUs
(costing thousands of dollars per hour), whereas our GNN-based model can make
an inference within a second with one GPU.

Figure 5.  A code summarization model for the deep-learning-based
documentation generation approach via GNN. There are three steps of data pre-
processing (1) We first extract text code pairs from existing notebooks. (2)
We generate AST from code. (3) We tokenized each word and translated them into
embeddings. And (4), the GNN model architecture.

In order to fine-tune the model, we constructed a training dataset for our
particular context. We collected the top 10% highly-voted notebooks from two
popular Kaggle competitions – House Price Prediction and Titanic Survival
Prediction (N = 1158). For each of the notebook, we first extracted code cells
and the markdown cells adjacent above as a pair of input and output (similar
to the data collection approach in (Agashe et al., 2019)). If there is an
inline comment in the first line of the code cell, we replaced the output of
the pair using the inline comment. In total, our dataset has 5,912 pairs of
code and its corresponding documentation. Following the best practice of model
training, we split the dataset into training, testing, and validation subsets
with an 8 to 1 to 1 ratio.

Before feeding data into the training process, we have a three-step pre-
processing stage, as illustrated in Figure 5. Step 1 removes the style
decoration, formats, and special characters that are not in Python grammar
(e.g., Notebook Magics). We also generate an AST for the source code input in
step 2 with Python AST library888https://docs.python.org/3/library/ast.html.
The AST result is equivalent to the source code but with more contextual and
relational information. In step 3, we tokenize source code to a sequence of
tokens with an input dictionary, and parse the AST nodes as a sequence of
tokens with the same input dictionary. We parse the relationship between AST
nodes as a matrix of edges. Finally, we tokenize the output documentation as a
sequence of tokens with a separate output dictionary. After this process, all
the tokens are transformed into an array of word embeddings — vectors of real
numbers. We use these data to train the network for 100 epochs, 30 batch
sizes, and 15 early stop points on a two Tesla V100 GPU cluster. Out of all
the epochs, we selected the model with the highest validation accuracy score.

To evaluate our model’s performance against baseline models, we conducted both
quantitative and qualitative evaluations, as suggested by (Ribeiro et al.,
2020). For the automated quantitative evaluation, we use BLEU scores (Papineni
et al., 2002) as the model performance metric. BLEU scores are commonly used
in the source code summarization tasks. It evaluates the word similarity
between the generated text and the ground truth text. We selected and trained
Code2Seq model (Alon et al., 2018) and Graph2Seq model (Xu et al., 2018) with
the same data split.

Our model achieves 11.41 (BLEU–a), which outperforms the baseline models
Code2Seq (BLEU–a = 9.61) and Graph2Seq (BLEU–a = 11.05). These scores suggest
that the data science documentation task is more difficult than the benchmark
code summarization tasks in the software engineering field. For example, in
data science, a notebook code cell can contain multiple code snippets and
functions.

Table 3.  Example output from the model. (Example A) The generated text well
describes the code. (Example B) The generated text vaguely describes the code.
(Example C) The generated text is poorly readable, but still captures the
keywords of the descriptions.

Example |  Code Cell |  Output From the Model
---|---|---
Example A |  python train = pd.read_csv(’./house-input/train.csv’) test = pd.read_csv(’./house-input/test.csv’)  |  Read the data
Example B |  python all_data = pd.get_dummies(all_data)  |  Convert all the data
Example C |  python pred = Tree_model.predict(x_test) pred = pd.DataFrame(pred) pred.columns = [”ConfirmedCases_prediction”]  |  Predicate to use a predict function for tests

In addition to the automated quantitative evaluation, we also conduct a
qualitative analysis of the generated documentation pieces. We found that
despite the word-to-word similarity score is low, the general quality of the
content is reasonable and satisfying for building a prototype system. As an
illustration, we provide three examples with both input and model generated
outputs, as shown in Table 4.3.1. In the Appendix, we provide full code cells
and model-generated outputs for the two experimental notebooks that we used in
the user study.

####  4.3.2. Query-Based Approach

Our formative study indicates that the well-documented Kaggle notebooks often
have the description of frequently-used data science code functions for
educational purposes. And sometimes data scientists directly paste in a link
or a reference to the external API documentation for a code function. Thus, we
implement a query-based approach that curates a list of APIs from commonly
used data science packages, and the short descriptions from external
documentation sites. In our system, we only cover
Pandas999https://pandas.pydata.org/docs/reference/index.html,
Numpy101010https://numpy.org/doc/stable/reference/, and Scikit-
learn111111https://scikit-learn.org/stable/modules/classes.html these three
libraries as a starting point to explore this approach. We argue that it can
be easily expanded to include other packages in the future. We collected both
the API names and the short descriptions by building a crawling script with
Python. When users trigger this query-based approach for a code cell, Themisto
matches the API names with the code snippets and concatenate all the
corresponding descriptions.

####  4.3.3. Prompt-Based Approach

Lastly, the system also provides a prompt-based approach that allows users to
manually create the documentation. Because our formative study found that a
well-documented notebook not only documents the process of the code, but also
interprets the output, and explains rationales. These types of documentation
are hard to generate with automated solutions To achieve it, we implement a
prompt-based approach. It detects whether the code cell has a cell output or
not: if the cell outputs a result, Themisto assumes that the user is more
likely to add interpretation for the output result, thus the corresponding
prompt will be inserted below the code cell. Otherwise, the system assumes the
user may want to insert a reason or some educational types of documentations,
thus it changes its prompt message.

##  5\. User Evaluation of Themisto

To evaluate the usability of Themisto and its effectiveness in supporting data
scientists to create documention in notebooks, we conducted a within-subject
controlled experiment with 24 data scientists. The task is to add
documentation to the given notebook. And each participant is asked to finish
two sessions, one with the Themisto support and one without its support. The
evaluation aims to understand (1) how well Themisto can facilitate documenting
notebooks and (2) how data scientists perceive the three approaches that are
used by Themisto for generating documentation.

###  5.1. Participants

We recruited 24 data science professionals as our evaluation participants in a
multinational IT company. We used a snowball sampling approach to recruit
participants, where we sent recruitment messages to friends and colleagues,
various internal mailing-lists, and Slack channels. We then asked participants
to refer their friends and colleagues. Our recruitment criteria are that the
participant must have had experience in data science projects and they are
familiar with Python and Jupyter Notebook environment. As shown in Table 4,
participants reported a diverse job role backgrounds, including expert data
scientists (N = 8), novice data scientists (N = 9), AI Operators or ML
engineers (N = 2), subject matter experts (N = 1), and application developer
(N = 4).

Table 4. Demographics of participants PID | Gender |  Job Role |  Work Experience in Data Science
---|---|---|---
P1 | M |  Expert Data Scientist |  5-10 years
P2 | M |  Application Developer |  3-5 years
P3 | M |  Novice Data Scientist |  less than 3 years
P4 | M |  Novice Data Scientist |  0 year (just start learning data science)
P5 | M |  AI Operator or ML Engineer |  3-5 years
P6 | M |  Novice Data Scientist |  less than 3 years
P7 | M |  Application Developer |  3-5 years
P8 | M |  Novice Data Scientist |  less than 3 years
P9 | F |  Expert Data Scientist |  3-5 years
P10 | M |  Expert Data Scientist |  5-10 years
P11 | M |  Expert Data Scientist |  more than 10 years
P12 | F |  Novice Data Scientist |  3-5 years
P13 | F |  Expert Data Scientist |  5-10 years
P14 | M |  Novice Data Scientist |  0 year (just start learning data science)
P15 | M |  Expert Data Scientist |  more than 10 years
P16 | M |  AI Operator or ML Engineer |  3-5 years
P17 | M |  Subject Matter Expert |  3-5 years
P18 | M |  Expert Data Scientist |  more than 10 years
P19 | F |  Application Developer |  3-5 years
P20 | F |  Expert Data Scientist |  3-5 years
P21 | M |  Novice Data Scientist |  3-5 years
P22 | M |  Novice Data Scientist |  less than 3 years
P23 | M |  Application Developer |  less than 3 years
P24 | M |  Novice Data Scientist |  less than 3 years

###  5.2. Study Protocol

We conducted a within-subject controlled experiment with 24 data scientist
participants. Their task was to add documentation to a given draft notebook,
which only has code and no documentation at all. The participants were told
that they were adding documentation for the purpose of sharing those
documented notebooks as tutorials for data science students who just started
learning data science. Each participant is asked to finish two sessions, one
with the Themisto support (Experiment condition) and one without its support
(Control Condition). We prepared two draft notebooks, one for each session,
shown in the Appendix.  The two experiment notebooks are adapted from winning
notebooks from two Kaggle challenges, which are not included in the model
training dataset: 1) House Price
Prediction121212https://www.kaggle.com/c/house-prices-advanced-regression-
techniques; 2) COVID Case
Prediction131313https://www.kaggle.com/c/covid19-global-forecasting-week-1.
The two notebooks have the same length (9 code cells) and a similar level of
difficulties. Although the two notebooks are simplified versions from winning
notebooks, they cover most stages in data science lifecycles. In addition, the
length of the notebooks falls into the middle range of the notebook length
distribution on the GitHub corpus (as refer to Figure 2). To counterbalance
the order effect, we randomized the order of the control condition and the
experiment condition for each participant, so some participants encountered
Themisto in their first session, some others experienced it in their second
session.

Each participant was given up to 12 minutes (720 seconds) to finish one
session.  We conducted three pilot run sessions, and all the three pilot
participants were able to finish a single task within 10 minutes, with or
without the support of Themisto. Before the experiment condition session, we
gave the participant a 1-minute quick demo on the functionality of Themisto.
All study sessions were conducted remotely via a teleconferencing tool. We
asked the participants to share their screen and we video recorded the entire
session with their permission. After finishing both sessions, we conducted a
post-experiment semi-structured interview session to ask about their
experience and feedback. We had a few pre-defined questions such as How do you
compare the experience of the documenting task with or without the support of
Themisto? or Did you notice the multiple candidates in the dropdown menu?
Which one do you like the most? In addition, participants were encouraged to
tell their stories and experience outside these structured questions. The
interview sections of the video recordings were transcripted into text.

###  5.3. Data Collection and Measurements

We have three data sources: the observational notes and video recording for
each session (N = 48), the final notebook artifact out of each session (N =
48), and the post-task questionnaire and interview transcripts (N = 24).

Our first group of measurements are from coding participants’ behavioral data
from the session recordings. In particular, we counted _the task completion
time_ (in secs) for all sessions. Then, for experiment condition only, we also
counted the followings: how many times a participant clicked on the light bulb
icon to check for suggestions (_code cells checked for suggestions_); how many
times a participant directly used the generated documentation (_markdown cells
created by Themisto_); how many times a participant ignored the generated
recommendations and manually created the documentation (_markdown cells
created by human_); and how many time a markdown cell is co-created by human
and Themisto(_markdown cells co-created by human and Themisto_). The result of
this analysis is reported in Table 5 and 6.

Secondly, to evaluate the quality of the final notebook artifact, we define
our second group of measurements by counting: the _number of added markdown
cells_ , and the _number of added words_ as these two are indicators of the
quantity and effort each participant spent in a notebook.  Also, we asked the
participants to give a self-reported satisfaction score to each of the two
documented notebooks. We considered that score (-2 to 2) as a self-reported
subject feeling of the notebook satisfaction. In parallel, we asked two
experts to rate the notebook-level quality (N = 48) with a 3-dimensional
rubric (based on (Garousi et al., 2013)) to evaluate the documentation’s
_readability, accuracy, and informativeness_ in a notebook. We considered
these three scores (-2 to 2) as an objective quality of the notebook.
Readability concerns whether the documentation is in readable English grammar
and words, while accuracy concerns how the documentation matches the code
content, and informativeness evaluates whether the documentation covers more
information units. Two experts iteratively discussed and evaluated the
notebooks until the independent ratings achieved an agreement ($\alpha$ =
0.76, Krippendorff’s alpha). The result of this analysis is reported in Table
5.

For the experiment session only, we conducted a cell-level expert rating (N =
194) using the same approach as in notebook-level expert rating. Two experts
iteratively discussed and evaluated the notebooks until the independent
ratings achieved an agreement ($\alpha$ = 0.88, Krippendorff’s alpha). The
result of this analysis is reported in Table 6. In addition, we asked the
participants to finish a post-experiment survey (5-point Likert Scale, -2 as
strongly disagree and 2 as strongly agree, Figure 6) to collect their feedback
specific on the system’s _usability, accuracy, trust, satisfaction, and
adoption propensity_ (based on (Weisz et al., 2019)).

Lastly, for the interview transcripts, four researchers of this research
project conduct an iterative open coding method to get the code, theme, and
representative quotes as the third group of data. They each independently
coded a subset of interview transcripts, and discussed the codes and themes
together. After the discussion, they when back and reiterated the coding
practice to apply the codes and themes to their assigned notebooks. Some
examples of the identified themes are: pros and cons of Themisto; preference
of the three document generation approaches; future adoption, and suggestions
for design improvement. We will report the qualitative results as supporting
materials together with reporting the quantitative results.

###  5.4. Results

Table 5.  Performance data in two conditions (M: mean, SD: standard deviation): the task completion time (secs), participants’ satisfaction with the final notebook (from -2 to 2), graded notebook quality, number of markdown cells, and number of words. In particular, participants spent less time to complete the task in the experimental condition than the control condition (p = .001); participants were more satisfied with the final notebook in the experimental condition than the control condition (p = .04). | Condition | M | SD |
---|---|---|---|---
Number of Added Markdown Cells | Experiment | 8.04 | 2.40 |
Control | 7.79 | 1.91
Number of Added Words | Experiment | 95.75 | 50.56 |
Control | 100.92 | 53.27
**Task Completion Time (secs) | Experiment | 391.12 | 200.15 |
Control | 494.75 | 184.28
*Satisfaction with the Final Notebook (-2 to 2) | Experiment | 0.96 | 0.69 |
Control | 0.54 | 0.83
Expert Rating: Accuracy (-2 to 2) | Experiment | 1.60 | 0.47 |
Control | 1.62 | 0.52
Expert Rating: Readability (-2 to 2) | Experiment | 0.65 | 0.83 |
Control | 0.90 | 0.57
Expert Rating: Informativeness (-2 to 2) | Experiment | 0.67 | 0.64 |
Control | 0.75 | 0.63

In this section, we present the user study results on: how Themisto improved
participants’ performance on the task, how participants perceived the
documentation generation methods in Themisto, and how participants described
the practical applicability of Themisto.

####  5.4.1. Themisto supports participants to easily add documentations to a
notebook.

Our experiment revealed that Themisto improved participants’ performance on
the task by reducing task completion time and improving the satisfaction with
the final notebooks.

We performed a two-way repeated measures ANOVA to examine the effect of the
two notebooks and the two conditions (with or without Themisto) on task
completion time. As shown in Table 5, participants spent significantly less
time (p¡.001) to complete the task using Themisto in the experiment condition
(M(SD) = 391.12 (200.15)) than in the control condition (M(SD) = 494.75
(184.28)). In addition, there was not a statistically significant effect of
notebooks on task completion time, nor a statistically significant interaction
between the effects of notebooks and conditions on completion time.

The post-experiment survey result supported our findings. Most participants
believed it was easier to accomplish the task with Themisto’s help (22 out of
24 rated agree or higher), as shown in Figure 6. And the Themisto generated
recommendation was accurate (20 out of 24 rated agree or higher).

Looking into the qualitative interview data, we can find some potential
explanations for why participants believed so. Participants reported that
Themisto provided them something to begin with, thus it was easier than
starting from scratch: The plugin makes it easy to just pick it and have
something simple. And then I got a couple of times where I went back and said,
Oh let me add a few more words. (P21).

Figure 6.  Results of the post-task questionnaire. Note that the disagrees are
not from the same participant.

####  5.4.2. Co-creation yields longer documentation and improves accuracy and
readability.

Through coding the video recordings for only the experiment-condition
sessions, we were able to examine the following questions: while the Themisto
was available, how did the participants use it? Did they check the
recommendations it generated? Did they actually use those recommendations in
their documentations added into notebooks?

As shown in Table 6, we found that while Themisto is available, for 86.11% of
code cells, participants checked the recommended documentation by clicking on
the light bulb icon to show the dropdown menu. Then, 46.90% of the created
markdown cells were directly adopted from Themisto’s recommendation; while
11.86% of the created markdown cells were manually crafted by humans alone.
The most interesting finding is that 41.24% markdown cells were co-created by
Themisto and human participants together: Themisto suggests a markdown cell,
human participants take it, and modify on top of it. This result suggested
that most participants used Themisto in the creation of documentation, and
some of them formed a small collaboration between humans and the AI. This
finding inspires us to further explore how participants co-create the
documentation with Themisto (Muller et al., 2021). By looking at the log data,
we discover several editing patterns. For example, many participants added
supplemental details (e.g., expanding the steps into substeps) to Themisto’s
suggested documentation. Participants also added stylistic edits, including
modifying document hierarchies, polishing sentences, and changing
conversational tones.

In order to explore the differences among documentation created by three
methods (created by Themisto only, co-created by human and Themisto, created
by human only), we conducted a cell-level expert rating (N = 194) along the
dimension of accuracy, readability, and informativeness. We also calculated
the word count of the documentation length. We performed a one-way ANOVA to
examine the differences among the three groups. As shown in Table 6, markdown
cells that are co-created by humans and Themisto have significantly more word
count (M(SD) = 15.45 (10.97)) than markdown cells that are manually written by
humans alone (M(SD) = 10.26 (7.41)) and the markdown cells that are directly
adopted from Themisto’s recommandation (M(SD) = 8.88 (7.14)), with F = 11.83,
p ¡ 0.001 . Markdown cells co-created by humans and Themisto also yield better
results in terms of accuracy (F = 9.43, p ¡ 0.001) and readability (F = 3.28,
p = 0.04), while for informativeness, there is no significant differences
across three groups. Our posthoc analysis suggested that no significant
differences were found between markdown cells created by Themisto and markdown
cells created by humans only along all dimensions (including word count,
accuracy, readability, and informativeness).

Table 6. Usage data of the plugin in experimental condition. The results indicate that participants used the plugin for recommended documentation on most code cells (86.11%). For markdown cells in the final notebooks, 46.90% were directly adopted from the plugin’s recommendation, while 41.24% were modified from the plugin’s recommendation and 11.86% were created by participants from scratch. |  $N$ |  $\%$ |  Word Count |  Accuracy |  Readability |  Informativeness
---|---|---|---|---|---|---
Code Cells Checked for Suggestions |  186 |  86.11 |  - |  - |  - |  -
Markdown Cells Created by Themisto Only |  91 |  46.90 |  8.88 (7.14) |  1.36 (0.54) |  1.94 (0.23) |  1.34 (0.56)
Markdown Cells Co-created by Humans and Themisto |  80 |  41.24 |  15.45 (10.97) |  1.68 (0.47) |  1.96 (0.17) |  1.51 (0.55)
Markdown Cells Created by Humans Only |  23 |  11.86 |  10.26 (7.41) |  1.28 (0.69) |  1.83 (0.36) |  1.35 (0.65)

####  5.4.3. Themisto increases participant’s satisfaction , while maintaining
a similar quality of the final notebook.

The post-task questionnaire revealed that participants were more satisfied
with the final notebook after using Themisto in the experiment condition than
in the control condition (p = .04) (Table 5). The interview results also
supported this finding. P14 believed that Themisto helped with wording:
Sometimes I knew what the cells were doing but I did not know how to put
things in a really good sentence for others.

Themisto also motivates participants to document the analysis details.
Although we did not see a difference in the number of markdown cells created
in two conditions or the number of words in total, Themisto helps them
overcome the procrastination of writing documentation and reminds them to
document things that they might ignore.

> I think I definitely overlooked some details when I was commenting without
> the tool, because I just made the assumption that people should know from
> the code… To be honest, I do not usually follow a good coding practice. My
> notebooks are really messy and I am the only person who can understand it. I
> feel sorry for anybody else that has to see it. (P19)

Moreover, participants believed that Themisto can help them form a better
documenting practice in the long term: It very useful to remind me to always
put some documentation in a timely manner. (P13).

The two experts’ gradings for the notebook quality suggest that there was not
a significant difference for the three dimensions of the quality rubric
(accuracy, readability, and completeness). In the post-task interview,
participants mentioned that the accuracy of the generated recommendations
plays a role in participant’s experience: My experience with the plugin is
definitely better. For the most part, the suggestions are pretty accurate.
Although sometimes I did make a few minor changes like rearranging the text.
(P5). Some participants also mentioned that they needed to edit the format of
the generated document to fit their context. We believe that while Themisto
offers convenience to improve the data scientists’ productivity and saves
their time, it may not provide the same level of readability as those
notebooks well articulated by humans. Thus, data scientists may want to
further revise the formatting and wording of the Themisto generated
documentation.

In summary, our experiment indicated that Themisto improves participants’
productivity for creating documentation. It also increases their perceived
satisfaction with the final notebook, compared to the notebooks written by
participants themselves.

####  5.4.4. The three approaches of generating documentation are suitable for
different scenarios.

In this section, we have an in-depth analysis of how participants perceived
the three different approaches that Themisto implemented to generate
documentations: the deep-learning-based approach, the query-based approach,
and the prompt-based approach. In the post-experiment interview, we explained
how Themisto generated the documentation with these three approaches, and
asked participants if they like or dislike one particular approach.

Participants reported that they felt the deep-learning-based approach provided
concise and general descriptions of the analysis process: I think the AI
suggestion gives me an overview. It is short, and has some useful keywords.
(P12).

Participants also suggested that the deep-learning-based approach sometimes
generated inaccurate or very vague documentation: The first one gives me a
very short summary, though it didn’t always say what the cell is doing. (P1).
But the deep-learning-based approach is still perceived useful. As it is short
and with only a couple of keywords, many participants believe it may be more
suitable for some quick and simple documentation task, or for the analyst
audience who can understand these short keywords.

In terms of the query-based approach, participants believed that the
documentation generated from this approach contains has longer and more
descriptive information. This approach is further perceived to be more
suitable for educational purposes: This one gives you really good information.
For some specific methods or calls, you don’t have to come up with a high-
level summary for others and you can directly use it. (P14).

Participants also acknowledged that such a query-based approach may not work
for some scenarios. For example, participants found that the query-based
approach was not useful for summarizing the very fundamental level data
manipulations, as there was no core API method in it. Some participants
mentioned that the usefulness of this query-based approach depends on the
audience.

> The [deep-learning-based approach] was really useful. The [query-based
> approach]… it depends on the audience. It is much more appropriate for a
> novice programmer. (P18)

We observed in the video recordings that participants rarely used the prompt-
based approach in the session. The interview data confirmed our speculation.
Some participants said that they liked the idea of user prompts, but they did
not use it because the deep-learning-based approach and the query-based
approach already gave them the actual content. Other participants pointed out
that the prompts were not intelligent enough, so they did not use it: It
always asks the same thing and I just ignored the prompts. (P18).

Participants suggested that the prompts could be designed to better fit the
context.

> Perhaps the system can infer what the code cell was doing [from the deep-
> learning-based approach], and show prompts accordingly. Like if I delete a
> data point from the dataset, there is a prompt asking why I considered it as
> an outlier or something. (P5)

Last but not the least, many participants preferred a hybrid approach to
combine the deep-learning-based approach and the query-based approach. For
example, P12 mentioned,

> The first one (deep-learning-based) tells me what the code cell is doing in
> general and the second one (query-based) tells me the details of the
> function. I would go with a hybrid approach. (P12)

####  5.4.5. Will participants use Themisto in their future data science
project?

Most participants indicated that they would like to use Themisto in the future
when answering the survey question as shown in Figure 6. The interview data
provides more detail and evidence to elaborate on this result. Participants
suggested various scenarios in which the Themisto could be useful in their
future work, such as they need to add documentation during the exploration
process for future selves, or they need to document a notebook in a post-hoc
way for sharing it with collaborators, or they need to mentor a team member
who is a novice data scientist, or they need to refactor an ill-documented
notebook written by others:

> When I am doing data analysis, I tend to write the code first because there
> is a flow in my head of what I need to do. And then I will go back afterward
> to use the plugin and add the comments needed. I will definitely do that
> before sharing that file or handing it over to others. (P12)

There was one participant who did not think Themisto could fit into his
workflow: I always write documentation before writing code. Maybe Themisto
does not work for people like me. (P3)

In our experiment, we provided the scenario as they were documenting the
notebook as a tutorial for some data science students. In the interview, we
asked how participants would document a notebook differently if they were
created documentation for the notebooks for non-technical domain experts
audience. Some participants suggested that computational notebooks may not be
a good medium to present the analysis to non-technical domain experts. They
would prefer to curate all the textual annotations in a standalone report or
slide decks. Some others believed that the notebook could work as the medium
but they would change the documentation by using less technical terminology,
adding more details on topics that the non-technical domain experts would be
interested in (e.g., how data is collected, potential bias in the analysis).

####  5.4.6. Participants suggest various design implications for automated
code documentation.

In the interview, participants provided various design suggestions to improve
Themisto and to design future technologies that can support data scientists to
document the notebook.

Participants expected Themisto to have more functionalities than simply
generating documentation for the code. For example, P13 proposed maybe
Themisto can also create a description to document the changes of versions and
the editing histories from different team members. P3 and P4 believed that the
automatic generation of Reason is very much needed for explaining decisions
such as why selecting a particular algorithm. P19 wanted the system to
automatically add explanations to the execution errors. Participants also
mentioned that Themisto should add more varieties into the generated content’s
formatting. They would like to see suggested documentation with a better
presentation.

And lastly, some participants suggested that maybe such a documentation
generation system can take consideration of the purpose of the notebook, the
domain-specific terminology, or the indivials’ habits for writing
documentation.

####  5.4.7. Summary of the Results

In summary, our study found that Themisto can support data scientists in
generating documentation by significantly reducing their time spent on the
task, and improving the perceived satisfaction level of the final notebook.
When Themisto is available, participants are very likely to check the
generated documentation as a reference. Many of them directly used the
generated documentation, a few of them still prefer to manually type the
documentation, while many of them adopted a human-AI co-creation approach that
they used the AI-generated one as a baseline and keep improving on top of it.
Participants perceived the documentation generated by the deep-learning-based
approach as a short and concise overview, the documentation generated by the
query-based approach as descriptive and useful for educational purposes, but
they rarely used the prompt-based approach. Overall, participants enjoyed
Themisto and would like to use it in the future for various documenting
purposes.

##  6\. Discussion

###  6.1. The Documentation Practices in Data Science is Different from in
Software Engineering

The practice of documentation in data science has both overlaps and strong
contrasts in relation to the ones in software engineering in many facets.
Software engineers write inline comments in their work-in-progress code to
help collaborators understand the behavior of the code without the burden of
going through thousands of lines of code; they document changes of their code
for better version management and improving awareness of their collaborators;
when others need to build upon their work, they write formal documentation and
Readme files to describe how to use functions and API in their packages or
services (Lowndes et al., 2017; Saltz et al., 2017; Zhang et al., 2020). Data
scientists write computational narratives as a practice of literate
programming (Kery and Myers, 2017; Park et al., 2018), and as a way to think
and explore alternatives. Thus, notebooks often have orphan code cells or out-
of-order code snippets, which leads to lower reusability of the notebook and
further highlights the importance of documentation in the notebook. As we
found in our formative study, well-documented notebooks explain more than the
behavior of the code. Notebooks cover various topics including describing and
interpreting the output of the code, explaining reasons for choosing certain
algorithms or models, educating the audience from different levels of
expertise, and so on.

Thus, many interventions and lessons learned about documentation in software
engineering may not apply in data science context. For example, how can we
evaluate the quality of the documentation? Software documentation can be
assessed based on attributes like completeness, organization, the relevance of
content, readability, and accuracy (Garousi et al., 2013). Our experiment
found that the quality scores assessed by these rubrics does not reflect
users’ satisfaction with the final notebooks. Despite many people’s efforts to
creating a standard documentation practice (Rule et al., 2019; Konkol et al.,
2020), it remains questionable whether there is a one-size-fits-all solution.
For example, Rule et al. (Rule et al., 2019) suggested ten rules for writing
and sharing computational analysis in Jupyter notebooks. The first rule they
proposed is to tell a story to the audience. However, this description is very
general as people may approach storytelling differently. As we observed in
Kaggle notebooks, some notebook authors prefer to use concise and accurate
language while others use more colloquial and creative language. These
creative notebooks stand out and receive many votes and compliments from the
Kaggle community. As we recognize documentation in data science as a fluid
activity, traditional template-based approaches to aid documentation writing
may not work in data science because they can not capture a broader aspects of
documentation, and limit the expressiveness of storytelling.

We argue that future work should recognize the difference between data science
and software engineering, and tailor the documentation experience for data
scientists. For example, Callisto (Wang et al., 2020b) harnessed the fact that
data scientists engage in synchronous work and discussion, and used contextual
links between discussion messages and notebook content to aid the explanation
of notebooks.

###  6.2. Human-AI Collaboration in Code Documentation in Data Science

We argue that AI-assisted code documentation process can be viewed as a co-
creative process in which machine learning fits into the human workflow and
collaborate together to create documents in a notebook.  The notion of a
“partnership relationship” between human data scientists and AI has been
discussion by Wang et al. (Wang et al., 2019), and is part of a larger
research discussion by many others (e.g., (Cai et al., 2019; Seeber et al.,
2020; Malone, 2018)).  We consider this partnership as broadly defined where
an AI system does not need an avartar or a conversational interface, but this
AI system should be designed to fit into the existing human workflow and
assist some parts of the human task to improve the quality or productivity.
Human-AI collaboration, as opposed to human-AI competition (portraited by
AlphaGo or DeepBlue), should be the ultimate goal of human-AI interaction
research. Various human-centered AI design principles (such as human-in-the-
loop) are means to get to this end goal. Our study demonstrated another means
to achieve human-AI collaboration, where we combined the fully automated
neural network approach and the less advanced rule-based or prompt-based
approaches. This design is to acknowledge the limitation of today’s neural-
network modeling. Our result showed that the combined human + AI effort
produced a satisfied level of quality at a _more rapid pace_ than what human
or AI could achieve alone.

We also observed the user interaction pattern in which the AI creates first
draft of the documentation, followed by human review and editing, resulted in
a final artifact that not only met the bar for quality, but exceeded it for
the level of satisfaction. Participants were happier with their code
documentation when they were assisted by the AI system to create it, rather
than when they worked alone. Thus, we conclude that the benefits of having an
AI partner in this task stem from being able to produce the same high level of
objective quality, but at a much more rapid pace (20% faster on average) and
with a higher level of satisfaction with the end product.

We speculate that one of the contributing factors for why people were
accepting of the AI’s suggestions is because the final decision of taking
those suggestions was up to the human.  As an alternative, we could have
designed the system to always automatically produce a markdown documentation
cell for each identified code cell, but we decided not to. Because this fully-
automated design is an extreme in the framework of automation put forth by
Parasuraman et al. ((Parasuraman et al., 2000); see also (Horvitz, 1999)),
which people may feel being replaced. Our results confirmed our assumption —
participants reported that they enjoyed being able to see multiple
suggestions, created using different algorithms, and select the one that was
the closest match to their intent in documenting a code cell. This level of
interaction corresponds to AI executes a selection only after a human has
approved in the Parasuraman et al. model (Parasuraman et al., 2000).

Our result also shed light on the research question in (Wang et al., 2019)
about the conditions under which human data scientists will enjoy working with
AI partnership . In our case, maintaining control of the initiative and the
final decision is an important aspect for people’s enjoyment and acceptance of
the AI system. It remains to be studied whether people prefer both to control
their own initiative and the initiative of a machine teammate, as proposed in
Shneiderman’s recent two-dimensional model (Shneiderman, 2020).  Also, we did
not focus on the explanability or trust aspect of the designed AI system, such
as how to visualize the connection between the generated documentaiton and the
original code. In the future, the explanability and trust aspects of the AI
system in the data science context is a very critical research topic (e.g.,
(Weidele et al., 2020; Drozdal et al., 2020)), and should also be prioritized
in the research agenda.

There are many other tasks in a data science project’s lifecycle that could
use AI’s help, such as model presentation or feature engineering (Wang et al.,
2021b). In the future, we plan to extend our work to design more human-
centered AI systems to support users in these data science tasks as well.

In the future, we plan to explore whether the identified benefits and
tradeoffs persist or not after a long period of adoption by users. One of the
potential benefits could be: the human-AI collaboration work style helps users
to learn more from the AI suggested/reminded documentations, thus they realize
more of the value of adding code documentation to notebooks; in contrary,
maybe users become over-reliance on AI systems thus they de-skill in this code
documentation task, both hypotheses await future research to evaluate.

###  6.3. Design Implications

We offer designers and tool builders the following suggestions to encourage
data scientists to write better documentation:

####  6.3.1. Towards Hybrid and Adapted Code Summarization

Our evaluation of Themisto indicates that instead of a fully automatic
approach, data scientists prefer to use a hybrid method for helping them write
documentation. We argue that future work for code summarization should
investigate a hybrid and adapted approach. We suggest that adaptive
interactive prompting may be a worthwhile research topic. For example, prompts
could be based on the contents of the code cell which the user was trying to
document. Another possibility is that prompts be based on the user’s own
history of writing markdown cells, and could either appeal to the user’s
strengths, or could anticipate and accommodate the user’s weaknesses. In a
more socially-oriented approach, users within an organization might rate the
initial set of prompts, voting some prompts up or down depending on their
usefulness. An evolution of this idea might allow users to propose new prompts
for use by selves and others (e.g., (DiMicco et al., 2009)). Furthermore, we
argue that future code summarization tools would benefit from a reinforcement
learning approach which learns from users’ modifications to the original
proposed texts, and could anticipate the users’ preference in subsequent
documentation.

####  6.3.2. Customizing the Recommendations based on Usage Scenarios

As prior work stated (Zhang et al., 2020), data science workers engage in
various collaborations during different stages of the data science lifecycle.
Documentation plays an important role in many scenarios. For example, handing
off work between data engineers, communicating results with stakeholders, or
informal notes to future self. Data science workers may have different needs
of the documentation for different usage scenarios. Designers and tool
builders should take a user-centered approach to understand the purpose of
documentation, the appropriate level of details, and the best way to present
the documentation. For example, participants suggested that future versions of
Themisto being able to document the changes of versions, errors, and related
online forum posts. Participants also suggested that they would like to see
more varieties into the generated narrative’s formatting.

####  6.3.3. Inverting Themisto – Automatic Code Generation from
Documentation.

The premise of Themisto was to generate descriptive material based on program
code. Following some of the ideas in Seeber et al. (Seeber et al., 2020), we
might invert this strategy. We recall that P3 told us that he wrote
documentation in advance of writing the code itself. If there are other people
who use the same discipline as P3, could we generate code from descriptive
text? We suspect that this idea would not work for just any textual
description. However, there could be certain stylized ways of writing
descriptions that might be translatable into code; pseudocode could provide a
starting point for the design of such a stylized type of description. We
recognize that this kind of approach would need to have a representation of
code packages and libraries, so that it could generate code that was
appropriately structured for those packages. Of course, package documentation
could be used to construct such a representation.

###  6.4. Limitation

Our formative study only explores notebooks from the Kaggle corpus, which may
leave out some varieties of markdown cells that only exist in messy notebooks
that can benefit from the support of documentation generation. However,
notebooks on the Kaggle platforms are based on real data and real problems,
and they aim for rich explanations and narratives, where other places do not
have high-quality notebooks with rich documentation. Future work should expand
the exploration on the other notebook corpus, for example, notebooks published
with scientific paper which contain fine-grained documentation.

Our experiment has several limitations: it focuses only on the documenting
(instead of coding) process, it is a controled experiment study, and
participants did not work on notebooks created by themselves. Thus, for
example, we do not know how participants would perceive the usefulness of the
tool in  realistic notebooks, which may be longer and more complicated (e.g.,
having out-of-order cells) than the notebooks we provided. However, we believe
the result is still promising to shed light for future research and future
system design. Future work can explore the generalizability of Themisto
through a long-term deployment study.

As for the Human-AI Collaboration research initiative, our work only reports
the findings on how human and AI collaborated at a coarse-grained level (Table
6). In the future work, we will have an indepth analysis to break down to the
level of individual cells, and further analyze the difference between
automatically-generated, co-edited, or manually-produced cells. This detailed
analysis will help us to understand how human behave and perceive the fine-
grained collaboraiton and interaction with the AI partner. And such findings
and its derived design insights could also help researchers who are studying
Human-AI Collaborations in other usage scenarios (e.g., in Healthcare or in
Educational settings (Xu et al., 2021)) beyond the nodebook documentation
context in this paper.

##  7\. Conclusion

In this paper, we have designed and built Themisto to support human data
scientists in the notebook documentation task. This researhc prototype also
serves as a prompt to explore the human-AI collaboration research agenda
within the automated notebook documentation user scenario. The system design
is driven by insights from previous literature, and also by a formative study
that analyzed 80 highly-voted Kaggle notebooks to understand how human data
scientists document notebooks. The follow-up user evaluation suggested that
the collaboration between data scientists and Themisto significantly reduced
task completion time and resulted in a final artifact that not only met the
bar of quality, but also exceed it for the level of satisfaction.

##  Appendix A Example of Documentation Generation in Themisto

Table 7.  Example notebook (House Prediction), documentation generation in Themisto, and user submitted documentation (T - Markdown Cells Created by Themisto Only, C - Markdown Ceells Co-created by Humans and Themisto, H - Markdown Cells Created by Humans Only). Source Code |  DL-Based |  Query-Based |  Prompt-Based |  P2
---|---|---|---|---
python import pandas as pd import numpy as np from sklearn.linear_model import LassoCV from sklearn.model_selection import cross_val_score  |  Importing libraries |  Pandas is for data manipulation and analysis; NumPy is a library for … |  This code cell is for _ _ _ _ _ |  Importing libraries (T)
python train = pd.read_csv(’train.csv’) test = pd.read_csv(’test.csv’)  |  Read the data |  Read a comma-separated values (csv) file into DataFrame; Return the first 5 rows. |  This code cell is for _ _ _ _ _ |  Read the data (T)
python train.head()  |  Let’s see the values |  Return the first 5 rows |  The table shows _ _ _ _ _ |  Return the first 5 rows. (defValue=5) (C)
python all_data = pd.concat(( train.loc[:,’SubClass’:’SaleCond’], test.loc[:,’SubClass’:’SaleCond’]))  |  A generator for feature |  Concatenate pandas objects along a particular axis with optional set logic along the other axes. |  This code cell is for _ _ _ _ _ |  Concat train and test col ”SaleCondition” (C)
python all_data = pd.get_dummies(all_data)  |  Convert all the data |  Convert categorical variable into dummy/indicator variables |  This code cell is for _ _ _ _ _ |  Convert categorical variable into dummy/indicator variables. (T)
python all_data = all_data .fillna(all_data.mean())  |  Check the missing values |  Fill NA/NaN values using the specified method |  This code cell is for _ _ _ _ _ |
python X_train = all_data[:train.shape[0]] X_test = all_data[train.shape[0]:] y = train.SalePrice  |  Create the target and the test data |  Slice string |  This code cell is for _ _ _ _ _ |
python model_lasso = LassoCV( alphas = [1, 0.1, 0.001, 0.0005]) .fit(X_train, y)  |  Model |  Lasso linear model with iterative fitting along a regularization path. |  This code cell is for _ _ _ _ _ |  Fit regression model (H)
python def rmse_cv(model): rmse= np.sqrt(-cross_val_score( model, X_train, y, scoring= ”neg_mean_squared_error”, cv = 5)) return(rmse) rmse_cv(model_lasso).mean()  |  A simple example model with the lasso |  Evaluate a score by cross-validation |  The result indicates that _ _ _ _ _ |  Define score function and evaluate (H)
Table 8.  Example notebook (Covid Prediction), documentation generation in Themisto, and user submitted documentation (T - Markdown Cells Created by Themisto Only, C - Markdown Ceells Co-created by Humans and Themisto, H - Markdown Cells Created by Humans Only). Source Code |  DL-Based |  Query-Based |  Prompt-Based |  P5
---|---|---|---|---
python import numpy as np import pandas as pd from sklearn.ensemble import RandomForestClassifier  |  Importing libraries |  Pandas is for data manipulation and analysis; NumPy is a library for … |  This code cell is for _ _ _ _ _ |  Importing libraries (T)
python train = pd.read_csv(”train.csv”) test = pd.read_csv(”test.csv”) train.head()  |  Read the data |  Read a comma-separated values (csv) file into DataFrame; Return the first 5 rows. |  The table shows _ _ _ _ _ |  Read and sanity check the data (C)
python train.describe()  |  Let’s see the values |  Generate descriptive statistics. Descriptive statistics include … |  The table shows _ _ _ _ _ |
python train[”Date”] = train[”Date”] .apply(lambda x: x.replace(”-”,””)) train[”Date”] = train[”Date”] .astype(int) train.head()  |  Convert all the data |  Replace a specified phrase with another specified phrase |  The table shows _ _ _ _ _ |  Preprocess the data (C)
python train.isnull().sum()  |  Check the missing values |  Detect missing values for an array-like object |  The result indicates that _ _ _ _ _ |  Check the missing values (T)
python test[”Date”] = test[”Date”] .apply(lambda x: x.replace(”-”,””)) test[”Date”] = test[”Date”] .astype(int)  |  Convert all the data |  Replace a specified phrase with another specified phrase |  This code cell is for _ _ _ _ _ |  Preprocess the date column (C)
python x = train[[’Lat’, ’Long’, ’Date’]] y = train[[’ConfirmedCases’]] x_test = test[[’Lat’, ’Long’, ’Date’]]  |  Create the target and the test data |  Select subsets of data |  This code cell is for _ _ _ _ _ |  Create the train/test data and the target (C)
python Tree_model = RandomForestClassifier( max_depth=200, random_state=0) Tree_model.fit(x,y)  |  Model |  A random forest is a meta estimator that fits a number of decision tree classifiers on … |  This code cell is for _ _ _ _ _ |  Define and configure the model A random forest is a meta … We also train the model with ‘.fit()‘ (C)
python pred = Tree_model.predict(x_test) pred = pd.DataFrame(pred) pred.columns = [”ConfirmedCases_prediction”]  |  Predicate to use a predicate function for tests |  A random forest is a meta estimator that fits a number of decision tree classifiers on … |  This code cell is for _ _ _ _ _ |  Run the model to generate predictions on the test data and store them as a ‘DataFrame‘(H)

##  Appendix B Coding Book for the Interview Transcripts

Table 9. Coding Book for the Interview Transcripts Theme |  Code
---|---
Pros of Themisto |  Easy to Use
Provie Inspirations
Improve Content
Efficiency
Hybrid Approach
Useful for Long Term
Prefer the Plugin
Cons of Themisto |  Inaccurate
Not Useful
Perceptions of the Deep-Learning-Based Approach |  Concise
Useful
Accurate
Inaccurate
For Own Use
For Collaboration Use
Perceptions of the Query-Based Approach |  Descriptive
Too Long
Useful
Confusing
Instructive
Perceptions of the Prompt-Based Approach |  Tedious
Easy to Use
Inspiring
Future Adoption |  Positive Adoption Propensity
Scenarios for Future Adoption
Negative Adoption Propensity
Design Improvements |  More Options Generated by AI
Handle Presentation and Formatting
Summarize Other Information (e.g., Reasons, Summary, Errors)
Custimization
Optimize UI
Adaptive Prompts

###### Acknowledgements.

We thank all of our participants for their help in the study, and the
annonymous reviewers for their valuable feedback.

## References

  * (1)
  * Agashe et al. (2019) Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. 2019\.  Juice: A large scale distantly supervised dataset for open domain context-based code generation.  _arXiv preprint arXiv:1910.02216_ (2019).
  * Alon et al. (2018) Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018\.  code2seq: Generating sequences from structured representations of code.  _arXiv preprint arXiv:1808.01400_ (2018).
  * Arnold et al. (2019) Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilovic̀, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al. 2019\.  FactSheets: Increasing trust in AI services through supplier’s declarations of conformity.  _IBM Journal of Research and Development_ 63, 4/5 (2019), 6–1.
  * Bai and Hu (2018) Liang Bai and Yanli Hu. 2018.  Problem-driven teaching activities for the capstone project course of data science. In _Proceedings of ACM Turing Celebration Conference-China_. 130–131.
  * Cai et al. (2019) Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. 2019\.  Human-centered tools for coping with imperfect algorithms during medical decision-making. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Chattopadhyay et al. (2020) Souti Chattopadhyay, Ishita Prasad, Austin Z Henley, Anita Sarma, and Titus Barik. 2020\.  What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * de Souza et al. (2005) Sergio Cozzetti B de Souza, Nicolas Anquetil, and Káthia M de Oliveira. 2005.  A study of the documentation essential to software maintenance. In _Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information_. 68–75.
  * DiMicco et al. (2009) Joan Morris DiMicco, Werner Geyer, David R Millen, Casey Dugan, and Beth Brownholtz. 2009\.  People sensemaking and relationship building on an enterprise social network site. In _2009 42nd Hawaii International Conference on System Sciences_. IEEE, 1–10.
  * Drozdal et al. (2020) Jaimie Drozdal, Justin Weisz, Dakuo Wang, Gaurav Dass, Bingsheng Yao, Changruo Zhao, Michael Muller, Lin Ju, and Hui Su. 2020.  Trust in automl: Exploring information needs for establishing trust in automated machine learning systems. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_. 297–307.
  * Eddy et al. (2013) Brian P Eddy, Jeffrey A Robinson, Nicholas A Kraft, and Jeffrey C Carver. 2013.  Evaluating source code summarization techniques: Replication and expansion. In _2013 21st International Conference on Program Comprehension (ICPC)_. IEEE, 13–22.
  * Fernandez-Bes et al. (2016) Jesus Fernandez-Bes, Jerónimo Arenas-García, and Jesús Cid-Sueiro. 2016.  Energy generation prediction: Lessons learned from the use of Kaggle in Machine Learning Course.  _Group_ 7, 8 (2016), 9.
  * Garousi et al. (2013) Golara Garousi, Vahid Garousi, Mahmoud Moussavi, Guenther Ruhe, and Brian Smith. 2013.  Evaluating usage and quality of technical software documentation: an empirical study. In _Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering_. 24–35.
  * Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018\.  Datasheets for datasets.  _arXiv preprint arXiv:1803.09010_ (2018).
  * Geiger et al. (2018) R Stuart Geiger, Nelle Varoquaux, Charlotte Mazel-Cabasse, and Chris Holdgraf. 2018.  The types, roles, and practices of documentation in data analytics open source software libraries.  _Computer Supported Cooperative Work (CSCW)_ 27, 3-6 (2018), 767–802.
  * Head et al. (2019) Andrew Head, Fred Hohman, Titus Barik, Steven M. Drucker, and Robert DeLine. 2019.  Managing Messes in Computational Notebooks. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. ACM, New York, NY, USA, Article 270, 12 pages.  https://doi.org/10.1145/3290605.3300500
  * Holland et al. (2018) Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018.  The dataset nutrition label: A framework to drive higher data quality standards.  _arXiv preprint arXiv:1805.03677_ (2018).
  * Horvath et al. (2019) Amber Horvath, Mariann Nagy, Finn Voichick, Mary Beth Kery, and Brad A Myers. 2019.  Methods for investigating mental models for learners of APIs. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–6.
  * Horvitz (1999) Eric Horvitz. 1999\.  Principles of mixed-initiative user interfaces. In _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_. 159–166.
  * Hu et al. (2018) Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018.  Deep code comment generation. In _2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_. IEEE, 200–20010.
  * Iyer et al. (2016a) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016a.  Summarizing Source Code using a Neural Attention Model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ (Berlin, Germany, 2016-08). Association for Computational Linguistics, 2073–2083.  https://doi.org/10.18653/v1/P16-1195
  * Iyer et al. (2016b) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016b.  Summarizing source code using a neural attention model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2073–2083.
  * Jupyter ([n.d.]) Project Jupyter. [n.d.].  _JupyterLab: the next generation of the Jupyter Notebook_.  https://blog.jupyter.org/jupyterlab-the-next-generation-of-the-jupyter-notebook-5c949dabea3
  * Jupyter (2015) Project Jupyter. 2015\.  Project Jupyter: Computational Narratives as the Engine of Collaborative Data Science.  Retrieved September 15, 2019 from https://blog.jupyter.org/project-jupyter-computational-narratives-as-the-engine-of-collaborative-data-science-2b5fb94c3c58.
  * Kajko-Mattsson (2005) Mira Kajko-Mattsson. 2005\.  A survey of documentation practice within corrective maintenance.  _Empirical Software Engineering_ 10, 1 (2005), 31–55.
  * Kery et al. (2019) Mary Beth Kery, Bonnie E. John, Patrick O’Flaherty, Amber Horvath, and Brad A. Myers. 2019.  Towards Effective Foraging by Data Scientists to Find Past Analysis Choices. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. ACM, New York, NY, USA, Article 92, 13 pages.  https://doi.org/10.1145/3290605.3300322
  * Kery and Myers (2017) Mary Beth Kery and Brad A. Myers. 2017.  Exploring exploratory programming. In _2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_ (2017-10). 25–29.  https://doi.org/10.1109/VLHCC.2017.8103446
  * Kery et al. (2018) Mary Beth Kery, Marissa Radensky, Mahima Arya, Bonnie E. John, and Brad A. Myers. 2018.  The Story in the Notebook: Exploratory Data Science Using a Literate Programming Tool. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. ACM, New York, NY, USA, Article 174, 11 pages.  https://doi.org/10.1145/3173574.3173748
  * Knuth (1984) Donald E. Knuth. 1984\.  Literate Programming.  _Comput. J._ 27, 2 (1984), 97–111.  https://doi.org/10.1093/comjnl/27.2.97
  * Konkol et al. (2020) Markus Konkol, Daniel Nüst, and Laura Goulier. 2020\.  Publishing computational research–A review of infrastructures for reproducible and transparent scholarly communication.  _arXiv preprint arXiv:2001.00484_ (2020).
  * Kross and Guo (2021) Sean Kross and Philip J Guo. 2021.  Orienting, Framing, Bridging, Magic, and Counseling: How Data Scientists Navigate the Outer Loop of Client Collaborations in Industry and Academia.  _arXiv preprint arXiv:2105.05849_ (2021).
  * Lau et al. (2020) Sam Lau, Ian Drosos, Julia M Markel, and Philip J Guo. 2020\.  The Design Space of Computational Notebooks: An Analysis of 60 Systems in Academia and Industry. In _2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. IEEE, 1–11.
  * LeClair et al. (2020) Alexander LeClair, Sakib Haque, Linfgei Wu, and Collin McMillan. 2020.  Improved code summarization via a graph neural network.  _arXiv preprint arXiv:2004.02843_ (2020).
  * Liu et al. (2019) Jiali Liu, Nadia Boukhelifa, and James R Eagan. 2019\.  Understanding the role of alternatives in data analysis practices.  _IEEE transactions on visualization and computer graphics_ 26, 1 (2019), 66–76.
  * Liu et al. (2020) Sijia Liu, Parikshit Ram, Deepak Vijaykeerthy, Djallel Bouneffouf, Gregory Bramble, Horst Samulowitz, Dakuo Wang, Andrew Conn, and Alexander Gray. 2020.  An ADMM based framework for automl pipeline configuration. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 4892–4899.
  * Liu et al. (2021) Xuye Liu, Dakuo Wang, April Yi Wang, Yufang Hou, and Lingfei Wu. 2021.  HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebooks. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Findings_.
  * Louie et al. (2020) Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai. 2020.  Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Lowndes et al. (2017) Julia S Stewart Lowndes, Benjamin D Best, Courtney Scarborough, Jamie C Afflerbach, Melanie R Frazier, Casey C O’Hara, Ning Jiang, and Benjamin S Halpern. 2017.  Our path to better science in less time using open data science tools.  _Nature ecology & evolution_ 1, 6 (2017), 1–7.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015\.  Effective approaches to attention-based neural machine translation.  _arXiv preprint arXiv:1508.04025_ (2015).
  * Maalej and Robillard (2013) Walid Maalej and Martin P Robillard. 2013.  Patterns of knowledge in API reference documentation.  _IEEE Transactions on Software Engineering_ 39, 9 (2013), 1264–1282.
  * Malone (2018) Thomas W Malone. 2018\.  How human-computer’Superminds’ are redefining the future of work.  _MIT Sloan Management Review_ 59, 4 (2018), 34–41.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting. In _Proceedings of the conference on fairness, accountability, and transparency_. 220–229.
  * Muller et al. (2019) Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, and Thomas Erickson. 2019\.  How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. ACM, New York, NY, USA, Article 126, 15 pages.  https://doi.org/10.1145/3290605.3300356
  * Muller et al. (2021) Michael Muller, April Yi Wang, Steven I. Ross, Justin D. Weisz, Mayank Agarwal, Kartik Talamadupula, Stephanie Houde, Fernando Martinez, John Richards, Jaimie Drozdal, Xuye Liu, David Piorkowski, and Dakuo Wang. 2021.  How Data Scientists Improve Generated Code Documentation in Jupyter Notebooks.
  * Oney and Brandt (2012) Stephen Oney and Joel Brandt. 2012.  Codelets: linking interactive documentation and example code in the editor. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 2697–2706.
  * Padioleau et al. (2009) Yoann Padioleau, Lin Tan, and Yuanyuan Zhou. 2009.  Listening to programmers—Taxonomies and characteristics of comments in operating system code. In _2009 IEEE 31st International Conference on Software Engineering_. IEEE, 331–341.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  BLEU: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. 311–318.
  * Parasuraman et al. (2000) Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. 2000\.  A model for types and levels of human interaction with automation.  _IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans_ 30, 3 (2000), 286–297.
  * Park et al. (2018) Soya Park, Amy X. Zhang, and David R. Karger. 2018.  Post-literate Programming: Linking Discussion and Code in Software Development Teams. In _The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings_ (Berlin, Germany) _(UIST ’18 Adjunct)_. ACM, New York, NY, USA, 51–53.  https://doi.org/10.1145/3266037.3266098
  * Perkel (2018) Jeffrey M. Perkel. 2018\.  Why Jupyter is data scientists’ computational notebook of choice.  _Nature_ 563 (2018), 145.  https://doi.org/10.1038/d41586-018-07196-1
  * Pimentel et al. (2016) João Felipe Pimentel, Saumen Dey, Timothy McPhillips, Khalid Belhajjame, David Koop, Leonardo Murta, Vanessa Braganholo, and Bertram Ludäscher. 2016.  Yin & Yang: demonstrating complementary provenance from noWorkflow & YesWorkflow. In _International Provenance and Annotation Workshop_. Springer, 161–165.
  * Piorkowski et al. (2021) David Piorkowski, Soya Park, April Yi Wang, Dakuo Wang, Michael Muller, and Felix Portnoy. 2021\.  How ai developers overcome communication challenges in a multidisciplinary team: A case study.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (2021), 1–25.
  * Rehman (2019) Mohammed Suhail Rehman. 2019\.  Towards Understanding Data Analysis Workflows using a Large Notebook Corpus. In _Proceedings of the 2019 International Conference on Management of Data_. 1841–1843.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.  _arXiv preprint arXiv:2005.04118_ (2020).
  * Roehm et al. (2012) Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. 2012.  How do professional developers comprehend software?. In _2012 34th International Conference on Software Engineering (ICSE)_. IEEE, 255–265.
  * Rule et al. (2019) Adam Rule, Amanda Birmingham, Cristal Zuniga, Ilkay Altintas, Shih-Cheng Huang, Rob Knight, Niema Moshiri, Mai H Nguyen, Sara Brin Rosenthal, Fernando Pérez, et al. 2019\.  Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks.
  * Rule et al. (2018a) Adam Rule, Ian Drosos, Aurélien Tabard, and James D. Hollan. 2018a.  Aiding Collaborative Reuse of Computational Notebooks with Annotated Cell Folding.  _Proc. ACM Hum.-Comput. Interact._ 2 (2018), 150:1–150:12.  Issue CSCW. https://doi.org/10.1145/3274419
  * Rule et al. (2018b) Adam Rule, Aurélien Tabard, and James D Hollan. 2018b.  Exploration and explanation in computational notebooks. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Saltz et al. (2017) Jeffrey Saltz, Kevin Crowston, et al. 2017\.  Comparing data science project management methodologies via a controlled experiment.  (2017).
  * Samuel and König-Ries (2018) Sheeba Samuel and Birgitta König-Ries. 2018.  ProvBook: Provenance-based Semantic Enrichment of Interactive Notebooks for Reproducibility.. In _International Semantic Web Conference (P &D/Industry/BlueSky)_.
  * Samuel and König-Ries (2020) Sheeba Samuel and Birgitta König-Ries. 2020.  ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks.  _arXiv preprint arXiv:2006.12110_ (2020).
  * Seeber et al. (2020) Isabella Seeber, Eva Bittner, Robert O Briggs, Triparna de Vreede, Gert-Jan De Vreede, Aaron Elkins, Ronald Maier, Alexander B Merz, Sarah Oeste-Reiß, Nils Randrup, et al. 2020\.  Machines as teammates: A research agenda on AI in team collaboration.  _Information & management_ 57, 2 (2020), 103174\.
  * Shi et al. (2011) Lin Shi, Hao Zhong, Tao Xie, and Mingshu Li. 2011\.  An empirical study on evolution of API documentation. In _International Conference on Fundamental Approaches To Software Engineering_. Springer, 416–431.
  * Shneiderman (2020) Ben Shneiderman. 2020\.  Human-centered artificial intelligence: Reliable, safe & trustworthy.  _International Journal of Human–Computer Interaction_ 36, 6 (2020), 495–504.
  * Sridhara et al. (2010) Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K Vijay-Shanker. 2010.  Towards automatically generating summary comments for java methods. In _Proceedings of the IEEE/ACM international conference on Automated software engineering_. 43–52.
  * Wang et al. (2020b) April Yi Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2020b.  Callisto: Capturing the “Why” by Connecting Conversations with Computational Narratives. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. ACM.
  * Wang (2016) Dakuo Wang. 2016\.  How people write together now: Exploring and supporting today’s computer-supported collaborative writing. In _Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion_. ACM, 175–179.
  * Wang et al. (2021a) Dakuo Wang, Josh Andres, Justin Weisz, Erick Oduor, and Casey Dugan. 2021a.  AutoDS: Towards Human-Centered Automation of Data Science.  _Proceeding of CHI’21_ (2021).
  * Wang et al. (2021b) Dakuo Wang, Q. Vera Liao, Yunfeng Zhang, Udayan Khurana, Horst Samulowitz, Soya Park, Michael Muller, and Lisa Amini. 2021b.  How Much Automation Does a Data Scientist Want?. In _preprint_.
  * Wang et al. (2020a) Dakuo Wang, Parikshit Ram, Daniel Karl I Weidele, Sijia Liu, Michael Muller, Justin D Weisz, Abel Valente, Arunima Chaudhary, Dustin Torres, Horst Samulowitz, et al. 2020a.  Autoai: Automating the end-to-end ai lifecycle with humans-in-the-loop. In _Proceedings of the 25th International Conference on Intelligent User Interfaces Companion_. 77–78.
  * Wang et al. (2019) Dakuo Wang, Justin D Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019.  Human-AI Collaboration in Data Science: Exploring Data Scientists’ Perceptions of Automated AI.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–24.
  * Weidele et al. (2020) Daniel Karl I Weidele, Justin D Weisz, Erick Oduor, Michael Muller, Josh Andres, Alexander Gray, and Dakuo Wang. 2020\.  AutoAIViz: opening the blackbox of automated artificial intelligence with conditional parallel coordinates. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_. 308–312.
  * Weisz et al. (2019) Justin D Weisz, Mohit Jain, Narendra Nath Joshi, James Johnson, and Ingrid Lange. 2019\.  BigBlueBot: teaching strategies for successful human-agent interactions. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_. 448–459.
  * Wenskovitch et al. (2019) John Wenskovitch, Jian Zhao, Scott Carter, Matthew Cooper, and Chris North. 2019.  Albireo: An Interactive Tool for Visually Summarizing Computational Notebook Structure. In _2019 IEEE Visualization in Data Science (VDS)_. IEEE, 1–10.
  * Wood et al. (2018) Jo Wood, Alexander Kachkaev, and Jason Dykes. 2018\.  Design exposition with literate visualization.  _IEEE transactions on visualization and computer graphics_ 25, 1 (2018), 759–768.
  * Xu et al. (2018) Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, Michael Witbrock, and Vadim Sheinin. 2018\.  Graph2seq: Graph to sequence learning with attention-based neural networks.  _arXiv preprint arXiv:1804.00823_ (2018).
  * Xu et al. (2021) Ying Xu, Dakuo Wang, Penelope Collins, Hyelim Lee, and Mark Warschauer. 2021.  Same benefits, different communication patterns: Comparing Children’s reading with a conversational agent vs. a human partner.  _Computers & Education_ 161 (2021), 104059.
  * Zhang et al. (2020) Amy X Zhang, Michael Muller, and Dakuo Wang. 2020\.  How do Data Science Workers Collaborate? Roles, Workflows, and Tools.  _arXiv preprint arXiv:2001.06684_ (2020).

  *[API]: Application Programming Interface
  *[HCI]: Human-Computer Interaction
  *[ML]: Machine Learning
  *[APIs]: Application Programming Interface
  *[NLP]: Natural Language Processing
  *[NMT]: Neural Machine Translation
  *[AST]: Abstract Syntax Tree
  *[AI]: Artificial Intelligence
  *[UI]: User Interface
  *[HTTP]: Hypertext Transfer Protocol
  *[GNN]: Graph Neural Network