# PromptRPA: Generating Robotic Process Automation on Smartphones from Textual
Prompts

Tian Huang<EMAIL_ADDRESS>0009-0007-8639-5929 Department of
Computer Science and Technology, Tsinghua UniversityBeijingChina ,  Chun Yu
0000-0003-2591-7993 Department of Computer Science and Technology, Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Weinan Shi
0000-0002-1351-9034 Department of Computer Science and Technology, Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Zijian Peng
0009-0005-6971-0064 Department of Computer Science and Technology, Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  David Yang
0000-0001-8260-3830 Department of Computer Science and Technology, Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Weiqi Sun
0009-0009-0957-4864 Department of Public Administration, Sichuan
UniversitySichuanChina<EMAIL_ADDRESS>and  Yuanchun Shi
0000-0003-2273-6927 Department of Computer Science and Technology, Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>
###### Abstract.

Robotic Process Automation (RPA) offers a valuable solution for efficiently
automating tasks on the graphical user interface (GUI), by emulating human
interactions, without modifying existing code. However, its broader adoption
is constrained by the need for expertise in both scripting languages and
workflow design. To address this challenge, we present PromptRPA, a system
designed to comprehend various task-related textual prompts (e.g., goals,
procedures), thereby generating and performing corresponding RPA tasks.
PromptRPA incorporates a suite of intelligent agents that mimic human
cognitive functions, specializing in interpreting user intent, managing
external information for RPA generation, and executing operations on
smartphones. The agents can learn from user feedback and continuously improve
their performance based on the accumulated knowledge. Experimental results
indicated a performance jump from a 22.28% success rate in the baseline to
95.21% with PromptRPA, requiring an average of 1.66 user interventions for
each new task. PromptRPA presents promising applications in fields such as
tutorial creation, smart assistance, and customer service.

Robotic Process Automation, UI navigation, natural language, UI understanding

††copyright: none††ccs: Human-centered computing Ubiquitous and mobile
computing systems and tools††ccs: Human-centered computing Graphical user
interfaces††ccs: Human-centered computing Interactive systems and tools Figure
1.  The workflow of PromptRPA. Blue arrows guide the generation process from
textual prompts to RPA, while green arrows indicate the knowledge accumulation
that occurs through user interactions, thereby enhancing the future
performance of the agents.

##  1\. Introduction

Robotic Process Automation (RPA) has emerged as a pivotal technology,
significantly boosting productivity and operational efficiency by automating a
wide range of tasks across various applications (Chakraborti et al., 2020).
Its ability to simulate human interactions on the graphical user interface
(GUI) allows for broad compatibility with third-party applications, without
necessitating modifications to existing code (Jovanović et al., 2018).
However, RPA primarily relies on predefined operation sequences (Ribeiro et
al., 2021; Herm et al., 2023). Customizing these sequences often requires
specialized skills in scripting languages and workflow design, which is a
significant barrier to adoption, particularly for users without a technical
background (Aguirre and Rodriguez, 2017). Additionally, the fixed operation
sequences have difficulty adapting to the frequently updated GUI, making them
susceptible to becoming quickly outdated.

To address this gap, we introduce PromptRPA, which employs a text-prompt-
driven RPA generation approach implemented on smartphones. PromptRPA is
capable of accepting unrestricted textual descriptions for automation
tasks—from concrete operational instructions like “open settings, locate WLAN,
and…” to high-level functional requests such as “change the ringtone”.
PromptRPA then automatically translates these textual prompts into executable
operation sequences on GUI, freeing users from the complexity of designing
automation scripts themselves.

PromptRPA serves as an intelligent navigation system on smartphones. It
interprets user intent from textual descriptions and matches them to its
internal “map” for route planning and decision-making, thereby automatically
generating RPA workflows. Should errors or issues arise, PromptRPA is designed
to adapt dynamically to ensure uninterrupted operation. Throughout the
process, PromptRPA faces several challenges. It needs to accommodate diverse
and partially articulated textual descriptions, which requires inferring
intentions from vague or incomplete instructions (Wen et al., 2001).
Additionally, it is challenging to deal with its incomplete “map”, which may
lack detailed route information for less common tasks or newly added
application functions (Islam et al., 2010). Another challenge is adapting to
frequent updates in interfaces and applications; as they evolve, the routes
PromptRPA relies on may also require updates (Venkatesh et al., 2022). These
challenges span across different stages of the process and require PromptRPA
to be highly flexible and adaptable.

To meet these challenges, we employ a multi-agent framework (Panait and Luke,
2005) for PromptRPA. The framework allocates specialized agents to sub-tasks
within different stages of the process, with each agent emulating specific
human cognitive functions. As illustrated in Figure 1, PromptRPA spans three
stages—from information collection and instruction generation to operation
mapping. Specifically, information collection involves gathering all necessary
descriptions of functions and steps for the RPA task. Instruction generation
entails creating formalized instructions to guide operations, and operation
mapping refers to translating text-based instructions into operations on
smartphones to ultimately complete the RPA task.

To support the multi-agent framework, two core elements are critical: a
comprehensive knowledge base (Venkatesh et al., 2022), which is our smartphone
“map”, and stable decision-making (Zhong et al., 2021), which ensures the
agents’ efficient and collaborative task execution. Therefore, we create an
evolving knowledge base that expands as it accumulates data and user
interactions. We introduce three key agents to meet the specific knowledge
demands of RPA generation: the _Analysis Agent_ , which actively fills in
missing contextual parameters from user inputs; the _Retrieval Agent_ , which
sources online tutorials to guide task automation; and the _Mobile Semantics
Agent_ , which conducts in-depth semantic analysis of mobile GUI, for better
alignment with textual inputs. To bolster decision-making, the _Assessment
Agent_ scrutinizes decisions made by other agents, serving as an internal
audit mechanism to improve accuracy and reliability. Beyond internal audit,
our agents are seamlessly integrated into a user interface, allowing users to
oversee and enhance decisions. User usage and intervention create a feedback
loop for continuous knowledge accumulation and system improvement, thereby
optimizing efficiency for future tasks.

To evaluate PromptRPA, we conducted a pipeline evaluation with 2,500 textual
prompts crowd-sourced. The experiment demonstrated that PromptRPA increased
the task success rate from a baseline of 22.28% to 95.21%, requiring only an
average of 1.66 user interventions for each new task. Moreover, it was proved
that the accumulation of user usage and intervention improved system
performance. Furthermore, a user study involving both skilled and unskilled
smartphone users confirmed PromptRPA’s effectiveness in assistance and its low
interaction cost. These results substantiate PromptRPA’s feasibility and
usability in practical settings.

Specifically, we propose PromptRPA, a multi-agent system accepting
unrestricted textual expression of task requirements, supplementing with
external knowledge primarily from online tutorials, and thus able to complete
a wider range of RPA tasks. By presenting the results of each stage, PromptRPA
supports more intuitive user interactions including dialogues, edits, and
demonstrations. It effectively reduces user burden while ensuring a high
success rate of RPA tasks. Due to the low-cost translation from text to RPA,
PromptRPA has promising applications in generating interactive tutorials,
enhancing smart assistance, and improving online customer service.

##  2\. Related Work

Our work intersects with four research areas: prior RPA systems, the knowledge
required for RPA, LLM-driven multi-agent frameworks, and optimizing the system
based on user feedback.

###  2.1. Prior RPA Systems

Prior RPA systems on smartphones typically work by replaying predefined
interaction flows on graphical user interfaces, thereby completing the
corresponding tasks. Early systems like SUGILITE (Li et al., 2017) and
APPINITE (Li et al., 2018) utilized programming by demonstration to enable
users to custom automation flows. However, for complex or unfamiliar
functions, creating flows is challenging, especially for beginners and the
elderly, who may lack essential task execution skills (Chen et al., 2017; Jin
et al., 2022). Subsequent developments introduce interactive, context-aware
tutorials for GUI applications (Fraser et al., 2019; Pongnumkul et al., 2011;
Li et al., 2015; Zhong et al., 2021; Li, 2021). However, these systems
primarily rely on pre-configured flows by developers (Wang et al., 2014), and
accommodating new tasks requires users to have specialized skills in script
design (Wewerka et al., 2020). Moreover, these fixed flows become outdated
when interface updates (Wu et al., 2023).

Recent efforts have focused on translating natural language into operation
sequences on smartphones. Notable contributions include Branavan et al.
(Branavan et al., 2009) and Li et al. (Li et al., 2020a). However, they
require detailed instructions for each step, necessitating that users have a
thorough understanding of task execution. While there are systems that support
general task requests, limitations exist, such as UGIF’s reliance on a pre-
collected tutorial dataset (Venkatesh et al., 2022) and Voicify’s dependence
on app developers’ implementation of deep links (Vu et al., 2023), failing to
meet a wide range of user needs. Additionally, they often function as a single
“black box”, making intervention or correction difficult for users when errors
occur (Venkatesh et al., 2022).

For easier use, our system is designed to accept unrestricted text inputs
related to the task, without requiring users to have prior knowledge of task
execution. By integrating online tutorial data and enhancing real-time
analysis of the mobile context, our system more effectively adapts to diverse
user needs and evolving interfaces. Moreover, our system features a user-
centric, transparent design pipeline to facilitate low-effort user
intervention.

###  2.2. Knowledge Required for RPA

Successful automation of smartphone tasks demands a comprehensive
understanding of diverse knowledge domains. For the input, tutorial datasets
need to be sufficiently rich to support the diversity of user needs. Regarding
the output, a nuanced grasp of mobile semantics is required to ensure robust
execution on smartphones.

Addressing user input presents challenges in accuracy and scope. User inputs
are often vague (Jansen et al., 1998), requiring supplementary contextual
information for accurate recognition of user intent (Smith et al., 2006;
Grudin and Jacques, 2019). Inspired by these studies, our system necessitates
a deeper exploration of context information related to mobile usage. In terms
of scope, existing UI trace datasets such as Rico (Deka et al., 2017), UGIF-
DataSet (Venkatesh et al., 2022), and PixelHelp (Li et al., 2020a) are pre-
collected and thus failing to meet a wide array of dynamic user needs
(Beneteau et al., 2019). AutoVCI (Pan et al., 2022) and Kite (Li and Riva,
2018) can only recognize user intents limited to predefined UI trace
collections. Voicify (Vu et al., 2023) is dependent on the application
developer’s integration of deep links; applications with limited deep links
exhibit lower feature coverage. When using search engines or AI tools like GPT
for tutorial information, the results can be irrelevant, outdated (Venkatesh
et al., 2022), fabricated (Shen et al., 2023), or low-quality (Burns et al.,
2022).

The other focus is the understanding of mobile GUI semantics, which is crucial
for widget identification and context comprehension. A wealth of research
focuses on providing semantic text annotations for UI elements (Li et al.,
2020b; He et al., 2021; Chen et al., 2022b; Wang et al., 2023). Other research
efforts include generating full-page text descriptions (Wang et al., 2021),
image-to-text conversion (Lee et al., 2022), and GUI embedding (Li et al.,
2021). Translating GUI images and hierarchical information into natural
language descriptions without information loss is vital for operation mapping,
as it enables textual instructions to align with GUI operations.

To tackle these multifaceted knowledge demands, our system employs specialized
agents, each focusing on a unique task: the _Analysis Agent_ for understanding
user intent from various text inputs through supplementing context
information, the _Retrieval Agent_ for managing online tutorial data, and the
_Mobile Semantics Agent_ for extracting UI semantics. These agents
collaboratively build an open, accumulative knowledge base, enabling effective
task automation on smartphones.

###  2.3. LLM-Driven Multi-Agent Frameworks

Large Language Models (LLMs) have been widely applied in decision-making tasks
recently. For one thing, some studies focus on the reasoning capabilities of
LLMs. ReAct (Yao et al., 2022) investigates the generation of reasoning traces
and actions, and ExpeL (Zhao et al., 2023) delves into the realm of autonomous
experiential learning. For another, increasing works aimed at enhancing the
LLMs’ interactive capabilities with external environments. Toolformer (Schick
et al., 2023) enables LLMs to interact with APIs, whereas Palo et al. (Palo et
al., 2023) explores their role within reinforcement learning agents.
Additionally, many works extend the utility of LLMs across different
modalities. “Do As I Can, Not As I Say” (Ahn et al., 2022) introduces a
grounding mechanism, ProgPrompt (Singh et al., 2023) targets robotic task
planning, and VisualGPT (Chen et al., 2022a) examines multi-modal
adaptability.

Moreover, several works address the issue of multi-agent cooperation driven by
LLMs. MetaGPT (Hong et al., 2023) addresses illusion issues in complex tasks.
Other frameworks, such as those proposed by Du et al. (Du et al., 2023), Liang
et al. (Liang et al., 2023), Wei et al. (Wei et al., 2023) and Zhang et al.
(Zhang et al., 2023), explore various aspects ranging from collective
decision-making to divergent thinking.

Prior work has shown the value of multi-agent frameworks in solving complex
tasks. In this paper, the workflow of PromptRPA is divided into several sub-
tasks, each assigned to a specialized agent. By optimizing their performance
individually and fostering collaboration among them, these agents collectively
enhance the overall task execution.

###  2.4. Optimizing the System Based on User Feedback

Interactive machine learning (IML) (Fails and Olsen, 2003; Dudley and
Kristensson, 2018; Amershi et al., 2014) aims to achieve complementary
benefits by integrating human perception and intelligence with the
computational power of computers (Deng et al., 2020). The framework has seen
extensive application in fields requiring annotation (Sperrle et al., 2019),
classification (Berg et al., 2019; de Medeiros et al., 2020), and intricate
data analysis (Carney et al., 2020).

In line with the principles of IML, PromptRPA employs an ensemble of agents
that evolve through user feedback. To minimize the user’s cognitive load,
PromptRPA supports feedback ranging from chats to simple text edits and hands-
on demonstrations.

##  3\. From Textual Prompts to RPA

Users approach tasks with a spectrum of understanding. Some may only have a
conceptual need but lack the specifics on how to accomplish it, while others
might already be armed with key procedural steps and are looking to automate
them. In either case, there exists a gap between what users know or wish to
achieve and the realization of these tasks through automation. Textual prompts
serve as a vital bridge, which can express user intent and be transformed into
structured sequences of operations, thereby enabling task automation.

In this section, we explore how textual prompts transition into RPA tasks. We
introduce a formalized RPA task model to better understand this process. The
model outlines the entire lifecycle of an RPA task, from its start to its
completion, emphasizing the role of textual prompts in initiating and guiding
the automation process. We then analyze the challenges within the process and
outline the design principles that underpin our system’s architecture.

###  3.1. RPA Task Model

Robotic Process Automation (RPA) has historically been viewed as a tool
designed to perform automated tasks by simulating human interactions with
computer systems (Van der Aalst et al., 2018). This traditional definition
focused largely on mimicking pre-defined, rule-based actions on graphical user
interfaces.

Advances in natural language processing and human-centric design have heralded
a more dynamic, and context-aware era in RPA, guided by textual prompts. These
textual inputs are not only easily formulated but also offer valuable
information to the RPA system, such as the task’s goal, steps, related
parameters, and specific applications.

Figure 2. Components of an RPA task model.

To integrate text and the execution requirements of RPA, we propose a
formalized _RPA task model_ , illustrated in Figure 2. The model comprises the
following components:

  * •

Function Description ($F$): A segment of text in natural language that
represents the overarching objective the RPA aims to achieve.

  * •

Step Description ($s_{1:n}$): A list of textual items detailing individual
steps within the automation process, each specified in natural language.

  * •

Instruction List ($i_{1:n}$): It consists of formalized instructions, derived
from natural language text, encoded in a machine-friendly standard language to
direct the operation sequence.

  * •

Operation Sequence ($o_{1:n}$): An ordered set of actions that the RPA
performs, often dynamically adjusted based on the textual prompts and the
real-time operating environment.

###  3.2. The Generation of an RPA Task Model

Our primary objective is to generate an RPA task model from a textual prompt
$P$, which involves constructing the model’s components by incorporating
related knowledge. Figure 3 illustrates the foundational pipeline for model
generation, segmented into the following three stages.

Figure 3. The generation process of an RPA task model.

Information Collection: From the textual prompt $P$, we extract useful
information and underlying intentions. Incorporating external knowledge
including contextual parameters and tutorials, we prepare all relevant natural
language information: the model’s overarching function description $F$ and its
granular step description $s_{1:n}$.

Instruction Generation: Employing the collected information, we strategize the
task’s progression. Given the instruction definition, we convert the step
description into a coherent list of standardized instructions $i_{1:n}$.

Operation Mapping: Following the planned instructions and understanding the
context of the actual operating environment, we finalize the operations,
resulting in the generation of the operation sequence $o_{1:n}$.

###  3.3. Existing Challenges

There has been prior work on generating executable operation sequences from
text, covering parts or even the entire pipeline described above. However, to
our knowledge, the emphasis of existing solutions has largely been on
validating the feasibility of such conversions. There is room for improvement
in their practical effectiveness as observed in real-world application.

Firstly, real-world application faces the challenge of input and output
diversity. Users’ expressions vary greatly, influenced by their varying levels
of knowledge. Developers or skilled users can provide clear, standard
instructions for specific operations, while novices or elderly users often
struggle to articulate even basic functional requirements, sometimes only
having a vague concept of the desired outcome (Jin et al., 2022). Most current
research (Pasupat et al., 2018; Li et al., 2020a; Pan et al., 2022) supports
explicit, single-step commands, such as “click the confirm button” or “input
Hello in the edit box”, but falls short in handling more complex tasks like
“activate developer mode”, and are particularly challenged by ambiguous
requests like “change the appearance”. As for outputs, the operation sequence
is not fixed. As shown in Figure 3, the sequence depends not only on the
textual guidance in the RPA task model but also on the operating environment.
Factors like smartphone model, app version, and page state can all influence
the operation sequence. Taking Figure 4 for an example, when a user wants to
post a moment, varying page states may require different operations.

Figure 4. For the same textual prompt “post a new moment”, however, the page
state can affect the specific operations required.

Secondly, to our knowledge, the current machine learning models are not always
satisfying in terms of semantic recognition of GUI elements, both in accuracy
and efficiency. Despite advances in image recognition algorithms, the small
granularity and rich styles of GUI icons (Dixon et al., 2014), coupled with
their complex semantics (Passini et al., 2008), pose significant challenges.
For instance, subtle distinctions can carry different meanings, like
distinguishing between a checked and an unchecked checkbox or discerning a
progress bar from a scrollbar, which may appear similar. Moreover, the range
of semantic concepts that pre-trained classification models can recognize is
limited (Liu et al., 2018). Particularly on mobile pages with insufficient
accessibility support, where recognizable text is sparse and reliance on image
interpretation is high, semantic understanding of the page degrades (Xie et
al., 2020; White et al., 2019). Even state-of-the-art algorithms like
Pix2Struct (Lee et al., 2022), “fail to capture the meaning of a more complex
screen” when only pixel information is accessible. Moreover, semantic analysis
of a small icon within a rich context typically requires several seconds (Wang
et al., 2023), a duration that is impractical for efficient operation.

Furthermore, once users issue initial commands, they often find themselves
excluded from the subsequent execution process. In cases of task failure, as
the entire process is a “black box” to them, it remains unclear which part
failed or whether the users could have assisted (Castelvecchi, 2016; Guidotti
et al., 2018). Beyond failed tasks, human intervention is often crucial for
tasks requiring personalized input, such as selecting a flight or choosing a
playlist song. Requiring users to provide all the details in initial commands
can be cognitively overwhelming. Additionally, the inherent ambiguity in
natural language (Winograd, 1972) requires a mutual confirmation process
between the AI and the user, with users needing to communicate their
intentions clearly (Ni et al., 2020), and the AI having to reveal enough
working details to build user trust (Amershi et al., 2019), possibly
necessitating multiple interactions for clarification.

Lastly, existing work is not immediately ready for use out-of-the-box. In
practical use, the number of applications and their functions desired by users
reaches into the billions (Islam et al., 2010). Prior RPA systems developed
using closed tutorial collections, such as UGIF (Venkatesh et al., 2022),
PixelHelp (Li et al., 2020a), and AITW (Rawles et al., 2023), support only a
fraction of these functions. They become inadequate when faced with
unsupported functions, mismatched personalized parameters, or GUI updates. In
comparison, online tutorials, while not matching the quality of carefully
collected closed datasets, far exceed them in terms of quantity. Broadening
the scope of tutorial knowledge to include these online resources could better
fulfill the diverse functional demands of users.

###  3.4. Design Guideline

Building on the RPA generation process and the existing challenges we analyzed
detailed above, the system design is founded on several key principles aimed
at effectively transforming textual prompts into executable operations:

Foster Versatility: In the RPA generation process, versatility entails
accommodating a wide range of inputs and operating environments (outputs). For
inputs, the system should address uncertainties, incompleteness, form
diversity, and the broad scope of user needs. During _information collection_
, the system should supplement missing information in user expressions, either
through predictive analysis or by querying users. For outputs, challenges from
the operating environment like rapidly updated pages should be addressed.
Hence, during _operation mapping_ , the system requires a deeper understanding
of the real-time environment and the ability to make adaptive decisions, not
limited to pre-defined instructions.

Optimize Knowledge Utilization: Each stage demands its specific expertise,
which should be as exhaustive and open-ended as possible, rather than confined
to a predefined set. During _information collection_ , the system should
recognize the function needs and retrieve relevant external tutorials if
necessary. During _instruction generation_ , adding rules to convert text into
instructions enhances the extraction of instructive content and minimizes
information loss. During _operation mapping_ , supplementing with real-time
page semantics and app-specific information is crucial.

Incorporating User Involvement: While user oversight can identify issues and
help enhance system effectiveness, it is crucial to streamline interactions to
minimize user burden. A stage-wise generation process helps make the results
of each stage transparent and allows for user intervention with minimal effort
when necessary, such as through quick quality assessments of text, simple text
modifications, or demonstrating a single step with guidance.

Enable Iterative Improvement: System enhancement stems from both capability
improvement and knowledge accumulation. Capability improvement relies on
mechanisms and models that support iterative optimization. Knowledge
accumulation arises from rising usage or manually introduced knowledge
supplements. Additionally, addressing knowledge obsolescence is essential, a
common issue in outdated mobile tutorials.

##  4\. PromptRPA

Realizing the above design guidelines necessitates a nuanced approach to
address the multifaceted challenges during each stage of the RPA generation.
Therefore, the utility of a multi-agent system framework here becomes evident.
By assigning sub-tasks to specialized agents, the framework enhances the
system’s capability to handle complex tasks, ensuring a more comprehensive and
efficient task execution process.

To optimize the performance of each agent, we emphasize two core factors: the
comprehensiveness of the knowledge base and the stability of decision-making.
They provide each agent with a shared pool of knowledge and clear behavioral
guidelines, ensuring that they collaborate effectively to accomplish tasks. In
alignment with this, Figure 5 illustrates how the system is partitioned into
specialized agents and a unified knowledge base. We will delve into each in
the following subsections.

Figure 5. System design of PromptRPA. The blue solid lines indicate the main
flow of the RPA generation process, the grey dashed lines represent optional
user intervention methods, the purple dashed lines signify knowledge
accumulation after successful task execution, and the red solid line denotes
instances where the agent requests user intervention.

###  4.1. Multi-Agent Design

Our system design is based on the prior agent system framework (Panait and
Luke, 2005; Dorri et al., 2018). Drawing inspiration from human capabilities,
each agent within our system is endowed with the following components:

  * •

Memory: The place where past experiences and learned knowledge are stored.

  * •

Tools: Instruments or utilities the agent utilizes to process data or perform
tasks.

  * •

Strategy: The agent’s internal cognitive mechanism for decision-making.

  * •

Action: The tasks the agent performs based on its cognition.

Besides these intrinsic components, transparency and user oversight are
cornerstones of PromptRPA. The agents are integrated seamlessly with a user
interface, allowing users to observe, modify, or augment agent decisions. The
interface is not just for monitoring purposes but aims to foster human-AI
collaboration.

According to the generation process of an RPA task model detailed in Section
3.2, we have designed six distinct agents specifically tailored to address
these three stages.

  * •

Information Collection:

    * –

Analysis Agent: Extracts and analyzes information encapsulated in the textual
prompt to construct a complete function description $F$.

    * –

Retrieval Agent: Acquires external knowledge relevant to the function
description and merges it with the textual prompt to build an enriched step
description $s_{1:n}$.

  * •

Instruction Generation :

    * –

Parsing Agent: Transforms the collected information into a series of
instructions $i_{1:n}$.

  * •

Operation Mapping:

    * –

Grounding Agent: Predicts and eventually executes operations $o_{1:n}$ on the
smartphone based on the generated instructions.

    * –

Mobile Semantics Agent: Extracts valuable semantic information from the mobile
interface to aid in operation mapping.

    * –

Assessment Agent: To balance prediction accuracy and user effort, it reviews
the predicted operations, evaluating if human intervention is required.

Subsequently, we will introduce how each agent works.

####  4.1.1. Analysis Agent

The agent aims to extract task-related information from the textual prompts,
as outlined in Section 3.1. The prompts may include descriptions of the
function or specific steps in the model. The agent employs the LLM to
segregate the text into these two components.

The function descriptions in the prompts often vary in formality and may lack
essential context, such as specific application names or phone models. Figure
6 demonstrates how the agent interacts with users to refine these
descriptions. Utilizing the LLM, the agent successively adopts three key
strategies:

Figure 6. Three scenarios where the agent interacts with the user to refine
the function description.

  * •

Normalization: The agent rephrases the original function description for
clarity and precision.

  * •

Automated Context Enrichment: The agent utilizes the LLM to determine the
essential parameters needed for the RPA task. When parameters are absent in
the prompt, the agent consults the context library (detailed in Section 4.2.2)
to fetch relevant records as potential candidates.

  * •

Interactive Context Enrichment: For parameters lacking records in the context
library, the agent queries the user to obtain the missing information.

After refining and enriching the function description, the user confirms it
via interactive dialogue. The agent then bundles the function description,
along with step descriptions (if available), and forwards them to the
_Retrieval Agent_ for subsequent processing.

####  4.1.2. Retrieval Agent

The _Analysis Agent_ can ensure a clear function description, but it may fall
short when it comes to providing detailed step descriptions, which the prompt
might not always cover. Addressing this void, the _Retrieval Agent_ is
responsible for crafting a complete step description. For efficiency, the
agent initially adopts the following strategies:

  * •

If the user confirms that the step description in the prompt is complete, the
system advances to the next stage without additional information collection.

  * •

Otherwise, the agent searches the historical RPA repository (detailed in
Section 4.2.1) for matching historical RPA task models, using function
description as the keyword, with LaBSE (Feng et al., 2022) for comparison,
which is a state-of-art multilingual sentence embedding model. If a match is
found, the agent will use the matched RPA task model, which has been
previously used within our system and is deemed reliable.

If the user is dissatisfied or no match is found, the agent searches for
relevant tutorials as the step description by using search engines, with the
function description as a query. The tutorial texts from the fetched web pages
are refined using SimpRead111https://simpread.pro/’s rule library and
summarized by the LLM. Tutorials are then empirically ranked based on criteria
including their original search ranking, data source, and textual quality
assessed by the LLM. Notably, the alignment with the step description
extracted from the user’s prompt is also a factor to consider.

The system presents the top 5 highest-ranking tutorials to users,
automatically selecting the highest-ranked tutorial in cases where users do
not make an active choice. By this stage, both the function and step
description for the RPA are prepared in place.

####  4.1.3. Parsing Agent

The agent’s primary responsibility is to translate step descriptions into a
sequence of formalized instructions tailored for smartphone operations.
Building on previous work (Li et al., 2020a), the definition of instruction
has been enriched to include a more detailed object description. As shown in
Table 4.1.3, an instruction is represented as a tuple comprising the operation
type, associated parameters, and object descriptions, usually referencing a
widget on the page. The agent utilizes LLM for parsing, because of its
capacity to function effectively with minimal initial data and its adeptness
in handling flexible text types (e.g., natural language, several keywords). As
the number of cases increases, the instruction definition can be extended and
the agent can be progressively refined (further discussed in Section 4.2.3).
Users can validate the parsed results and, where necessary, can perform manual
corrections.

Table 1. Formalized definitions of instructions.

Operation | Parameter | Object
---|---|---
open | application name |  description of UI pages or widgets (text, position, color…)
click | times (default=1)
long click | time
switch | value (true, false)
edit | text
scroll | direction (up, down, …)
back | page (previous, homepage) |

####  4.1.4. Grounding Agent

The final stage aims to predict specific operations on a smartphone based on
provided instructions. Specifically, it identifies a UI widget on the
smartphone with the object description in the instruction and thus executes
the relevant operation.

Initially, the agent captures the current page’s layout hierarchy and
screenshot. Based on text and interactivity attributes, it filters potential
candidate widgets. Subsequently, to ensure an efficient and effective
prediction of widgets, the agent adopts a multi-tiered strategy:

Historical RPA invocation: If the provided instructions are recurring, the
agent consults the historical RPA repository (in Section 4.2.1) to identify
widgets used in the past. It then compares these widgets with the ones on the
current page, using heuristic rules that focus on their rich GUI attributes. A
widget is considered a match only if it exceeds a certain level of similarity.

New instruction-based grounding: For new instructions, the agent’s sole
reliance is on the object description in the instruction. The agent then
employs the following strategies:

  1. (1)

Strict text matching: The agent uses strict text matching to find a widget
closely resembling the object description. A preset similarity threshold must
be met to confirm a match.

  2. (2)

Textual semantic matching: The agent identifies potential widgets, focusing on
semantic similarity with the operation and object description in the
instruction. It leverages LLM to predict the widget most suited to execute the
given instruction.

  3. (3)

Exploratory mechanism: Challenges in identifying the operation widget can
emerge due to mismatches between the actual page and the provided
instructions, often a result of outdated or erroneous instructions, variable
initial states of smartphone screens, or unexpected changes on the page
(Figure 4). Consequently, the agent does not strictly adhere to the
instructions for executing operations. Instead, our exploratory mechanism
leverages a wealth of contextual information to guide the decision-making
process of the LLM, enabling it to infer the most probable current operation.
Furthermore, the mechanism allows the agent to make several trials on the
smartphone, reducing total reliance on provided instructions. Issues such as
compound instructions (Figure 7) and inconsistent starting points (Figure 7)
can be effectively addressed. As the agent persistently accrues knowledge and
insights, the contextual information becomes richer, solidifying the
foundation upon which the LLM bases its inferences (further discussed in
Section 4.2.4).

Figure 7. Two examples illustrating the effectiveness of the exploratory
mechanism. (a) The compound instruction _Click “Sign up”_ requires additional
instructions. The agent can supplement the necessary steps. (b) The initial
state doesn’t match the first instruction. The agent can skip unnecessary
instructions.

For the implementation details of each strategy, see Appendix A.1. After the
agent makes its widget prediction, users have the option to manually
demonstrate the correct operation on the mobile GUI if they find the
prediction to be inaccurate.

####  4.1.5. Mobile Semantics Agent

The agent complements the _Grounding Agent_ by enhancing its understanding of
mobile semantics, a critical factor for successful grounding. There is a
discernible heterogeneity between text-based instructions and the operations
on the mobile page. GUI contains structured information and images but often
lacks textual alignment with instructions, which is most evident when widgets,
particularly icons, are void of textual descriptions.

To bridge this divide, the agent enriches the mobile page semantically, using
a combination of layout hierarchy and screenshots to create textual
descriptions for the page and its widgets. We employ the screen2words model
(Wang et al., 2021) to generate full-page textual descriptions and a
combination of OCR and pix2struct (Lee et al., 2023) for widgets. Based on
these models, the agent is fine-tuned through continuous absorption of new
training data (further discussed in Section 4.2.1).

####  4.1.6. Assessment Agent

Figure 8. A case of decision uncertainty prompting user intervention. The
target object is described as “Account Avatar”. The agent, which relies
heavily on textual descriptions for predictions, inaccurately predicts
“Account” because the avatar widget lacks a precise textual description. Due
to low confidence in its prediction, the agent seeks user intervention.

The agent is introduced to proactively address potential pitfalls and
streamline decision-making. It ensures the _Grounding Agent_ remains focused
on its primary tasks without being overloaded with error detection. The agent
adopts three key strategies:

  * •

Evaluation of execution: It assesses whether operations executed up to the
present are reasonable, to estimate the success of the ongoing task. The
proactive assessment provides the _Grounding Agent_ with informed guidance on
whether to continue or retrace steps.

  * •

Redundancy Prevention: By marking historical attempts, the agent avoids
falling into repetitive cycles, thereby preventing endless loops during the
exploration.

  * •

Intelligent User Involvement: It gauges the confidence in current decisions to
determine whether user intervention is appropriate. Figure 8 illustrates a
scenario where the agent prompts user intervention due to decision
uncertainty. Similarly, Figure 1 showcases a situation where the instruction
“Click a ringtone you like” introduces an ambiguity that necessitates user
assistance.

The agent leverages the LLM for reasoning. Based on the contextual
information, such as the current page description, operation history, and
immediate instructions, the LLM decides whether to proceed with the current
operation or roll back, providing a confidence score for its decision. More
details are in Appendix A.2.

###  4.2. Accumulated Knowledge

The agents draw from an ever-expanding knowledge base, which undergoes
refinement as user usage and intervention accumulate. As a feedback mechanism,
the knowledge base not only boosts the agents’ learning but also enhances the
entire system’s efficiency and task success rates.

Based on the guidelines in Section 3.4, the four categories of knowledge
integral to PromptRPA are as follows:

  * •

_Historical RPA Repository_ stores successfully executed RPA tasks. The
validated tasks not only demonstrate a higher success rate in future runs but
also retain adjustments tailored to specific operating environments or update
previously outdated tutorials, minimizing the need for manual
reconfigurations.

  * •

_Context Library_ stores and continuously updates user-specific mobile
parameters. It helps the system supplement necessary details when faced with
incomplete user expressions.

  * •

_Instruction Set_ focuses on improving the system’s ability to extract
instructive information from text. Failed parsing examples become valuable
learning materials.

  * •

_Mobile Interaction Graph_ strengthens the system’s understanding of app
functionalities and navigation logic. It serves as an auxiliary routing tool,
helping agents to better predict the operations.

We will introduce them in detail and outline strategies for the continuous
improvement they bring to the system.

####  4.2.1. Historical RPA Repository

Upon successful execution, PromptRPA archives the RPA task model’s data,
according to the structure defined in Section 3.1. Given that the _Grounding
Agent_ may not always align operations with initial instructions, the
repository focuses on storing the final sequence of operations. These
sequences are then reverse-engineered into corresponding instructions and step
descriptions via simple mapping rules, as detailed in Appendix B.1.

The historical RPA repository yields several optimizations across the system:

  * •

The _Retrieval Agent_ will prioritize historical RPA task models, enhancing
data reliability and minimizing retrieval time from online sources.

  * •

The _Grounding Agent_ gains enhanced predictive capabilities. It can utilize a
richer feature set, incorporating layout hierarchies and image data alongside
text descriptions. Notably, the stored mapping between instructions and
operations acts as a learning resource for the agent, facilitating the fine-
tuning of its internal model. This repository-driven learning equips the
_Grounding Agent_ to more accurately identify widgets when similar operation
descriptions are encountered in the future.

  * •

The _Mobile Semantics Agent_ benefits from a richer set of textual
descriptions for pages and widgets, using them to fine-tune its models. The
agent archives images and their embeddings (Wang et al., 2021) from the
repository. By comparing new images with archived ones, the agent improves its
accuracy in generating textual descriptions for similar images encountered
later.

####  4.2.2. Context Library

Derived from a formative study involving 24 college students and 20,265 online
Q&A questions (detailed in Appendix B.2), the library classifies these
parameters into three categories based on acquisition methods:

  1. (1)

Automated System Capture: Parameters that are available from the mobile GUI,
like battery status or current application in focus, are automatically
collected by the system. Sensitive data is not stored.

  2. (2)

User Input: Device-specific information authorized or provided by the user is
also collected. It could include permissions granted (e.g., access to camera
or location), as well as other profile details (e.g., preferred language)
which users can actively input through forms presented by the system.

  3. (3)

User-Driven Updates: The library continuously updates based on user-system
interactions. For instance, the earphones mentioned by the user in a certain
use (Figure 10) will be collected into the library.

In the library, a unique set of parameter records is maintained for each user.
The _Analysis Agent_ starts by identifying the contextual parameters required
for the specific task. It then consults the library, prioritizing the user’s
own historical records over those of other users. If required parameters are
missing, the user is prompted to provide the missing details, ensuring the
task’s function description is completed.

Figure 9. An example of records in the context library.

Figure 10. An example of auto-completing the textual prompt.

Figure 10 showcases a sample set of these parameter records, which will be
dynamically updated based on user interaction. With the aid of the library, as
depicted in Figure 10, missing parameters for recurring textual prompts are
automatically supplemented, which users have the option to confirm.

####  4.2.3. Instruction Set

The _Parsing Agent_ is tasked with converting natural language step
descriptions into formalized instructions. While conceptually straightforward,
practical application introduces issues, including the emergence of new
operation, parameter, or object description types and synonyms or deviation
from the format in Table 4.1.3. To address these issues, the instruction set
is regularly updated to include misinterpreted cases and their corrections.
These cases can come from user manual corrections, but most are resolved by
instructions reverse-engineered from successful RPA tasks, as described in
Section 4.2.1. Continuous refinement of the instruction set aids in the
ongoing fine-tuning of the model within the _Parsing Agent_.

Another important issue is related to compound instructions (Figure 7). For
such cases, the instruction set records both the compound instruction and its
decomposed result. Hence, if a compound instruction reoccurs, the _Parsing
Agent_ will automatically decompose it.

####  4.2.4. Mobile Interaction Graph

In the graph, each node represents a visited mobile page, and edges link
interactive widgets to the pages they lead to (Deka et al., 2017). As the
system navigates more pages, the graph evolves dynamically. The graph enables
virtual validation of each instruction before its real-world execution on a
smartphone. If the edge already exists in the graph, it anticipates the future
pages that could be visited after the operation, thus enriching the contextual
understanding for the _Grounding Agent_ and the _Assessment Agent_ ,
facilitating more informed inferences (see Appendix A for details).

###  4.3. Implementation

The system is divided into two primary components: a server and a client. The
server handles information collection, instruction generation, and accumulated
knowledge. It is built using the Flask framework and crawls search results
from Baidu, Bing, and Google by simulating web browser actions. On the client
installed on the smartphone, the Accessibility Service API
222https://developer.android.com/reference/
android/accessibilityservice/AccessibilityService is used to obtain page
layout data, and MediaProjection
333https://developer.android.com/reference/android/media/projection/MediaProjection
is used to capture screenshots. The client triggers operations through the
Accessibility Service API on the smartphone. When user intervention is
required, an overlay is provided to monitor the user’s demonstrations. Actions
captured by this overlay are transferred to the native mobile GUI to trigger
the corresponding operations. The PromptRPA’s agents use the GPT-4 API
444https://platform.openai.com/docs/models/gpt-4 for their language model
needs and GPT-3.5 Turbo model for fine-tuned needs. All involved prompts are
shown in the supplemental files.

##  5\. Pipeline Performance Evaluation

This section assesses the performance of PromptRPA’s three-stage
pipeline—information collection, instruction generation, and operation
mapping—as well as the overall system performance.

###  5.1. Experiment Setup

PromptRPA is designed to fulfill a wide range of real-world needs through
textual prompts provided by users, which involve various applications and
built-in functionalities. Therefore, we specifically created a dataset for the
experiment.

Targeting Chinese speakers, we selected 10 Chinese applications (9
applications and 1 system) for evaluation, as shown in Table 2. Five of them
are top-ranked in terms of active users as of 2021, and four have rankings
beyond 300. The diverse selection enables us to evaluate PromptRPA’s
performance across different user activity levels and application
complexities. We constructed 100 ground-truth RPA task models by the following
steps:

  * •

We collected 37,987 questions related to these applications by crawling
several Q&A websites. The questions were then ranked by their click-through
rates, and we selected 5 common and 5 uncommon questions for each application,
resulting in a total of 100 questions. Each question was used to construct one
RPA task model and each was manually annotated with a ground-truth function
description.

  * •

For each model, we manually identified the highest-quality tutorial on
tutorial websites based on the given function description. This tutorial text
serves as the ground-truth step description for the model.

  * •

We utilized crowdsourcing to collect textual prompts that participants would
use to generate RPA tasks. In total, we collected 25 different textual prompts
for each RPA task, amounting to 2,500 prompts. Among them, 100 prompts
contained complete step descriptions, negating the need for further tutorial
retrieval.

  * •

According to the step description, we then manually annotated the ground-truth
instructions for each RPA task model.

  * •

Finally, the experiment was conducted on a HUAWEI Mate30. For each RPA task
model, we manually recorded the ground-truth operation sequence on this
device. It is crucial to note that these sequences are not annotated based on
ground-truth instructions, which might be impractical due to issues like
outdated tutorials.

Table 2. List of applications used and examples of tasks in the experiment. WeChat, QQ, TikTok, Beautiful Weather, Safe and Sound, and Film Encyclopedia were also selected for the user evaluation in Section 6. Application | Category | Activity Ranking | Task Examples
---|---|---|---
WeChat | Message | 1 | 1\. View WeChat Bean details
2\. Clear cache
QQ | Message | 2 | 1\. Share QQ screen
2\. Provide feedback on an issue
TikTok | Social Media | 3 | 1\. Cancel the monthly payment
2\. View TikTok shop
Alipay | Pay | 4 | 1\. Set the notification sound
2\. View Alipay points
Weibo | Social Media | 5 | 1\. Check creation earnings
2\. View corrected followers
Beautiful Weather | Weather | 366 | 1\. Update the weather every 3 hours
2\. Observe the weather in Beijing
Safe and Sound | Lifestyle | 392 | 1\. Set the shipping address
2\. Enable the night mode do not disturb
Snail Sleep | Lifestyle | 418 | 1\. Set the goal as stress relief
2\. View one’s own sleep talk
Film Encyclopedia | Entertainment | 671 | 1\. Join the user experience program
2\. Bind QQ with the Film Encyclopedia
HUAWEI System Tools | System | - | 1\. Turn off Pure Mode
2\. Hide photos

###  5.2. Experiment Method

In our experiment, we evaluated three configurations of our system:

  * •

Standard PromptRPA.

  * •

PromptRPA(AK): PromptRPA enhanced with accumulated knowledge.

  * •

PromptRPA(AK+UI): PromptRPA further enhanced with both accumulated knowledge
and user intervention.

The experiment assessed PromptRPA’s performance at individual stages and
across the entire pipeline. Notably, in a single stage where a task was not
fully completed, without user intervention, the knowledge that PromptRPA could
accumulate was quite limited. Therefore, PromptRPA(AK) was evaluated solely on
its entire pipeline performance. When evaluating the performance of individual
stages, we used ground-truth inputs, ensuring that any output deviation
stemmed exclusively from the current stage, thus eliminating errors from
preceding stages. Specifically, in the instruction generation stage, we input
100 ground-truth function and step descriptions; for the operation mapping
stage, we input 100 ground-truth instruction lists. In evaluating the
information collection stage and the entire pipeline, the input comprised
2,500 original textual prompts. To minimize the impact of input order on
knowledge accumulation, we randomly shuffled the inputs and conducted five
rounds of testing to calculate the average performance.

We selected appropriate baselines for comparison at both individual stages and
the entire process. For the information collection stage, manual queries on
Google, Bing, and Baidu served as the baseline. For instruction generation, we
used Seq2act (Li et al., 2020a) as the baseline. For the operation mapping
stage, we employed strict text matching as the baseline. In assessing the
entire pipeline, due to the lack of existing work supporting open-domain
mobile tasks, our baseline was instructions generated purely by GPT-4,
followed by strict text matching for operation mapping.

###  5.3. Experiment Results

The experiment results are organized based on the three stages within
PromptRPA as well as the performance of the entire pipeline.

####  5.3.1. Information Collection

This stage focuses on evaluating whether the system can successfully generate
a function description and step description based on a user-provided textual
prompt.

Among the 2,500 textual prompts, 100 contained complete step information and
required no further online retrieval. Users accepted the analyzed results for
97.00% of these prompts, with a few exceptions attributed to the LLM’s data
fabrication issues. The remaining 2,400 prompts were used to assess the top 5
tutorials retrieved by the system. The criteria for a ‘hit’ was whether users
deemed a tutorial matching the ground-truth function description, regardless
of on-phone executability. A tutorial at the top indicated that the system
could automatically select it, eliminating the need for user intervention.

Before the experiment, we benchmarked the search capabilities of Google, Bing,
and Baidu using 100 ground-truth function descriptions. As Table 3 shows,
Baidu outperformed the others, achieving a 100.00% Top-5 hit rate and a 99.00%
Top-1 rate. The results demonstrate that all tasks could be matched with
online tutorials, provided the descriptions are sufficiently standard and
accurate.

Table 3. Performance of various search engines in retrieving tutorials with ground-truth function descriptions of RPA task models. | Top-5 | Average | Top-1
---|---|---|---
| Hit Rate(%) | Rank | Hit Rate(%)
Google | 93.00 | 1.23 (sd=0.59) | 79.00
Bing | 96.00 | 1.25 (sd=0.70) | 81.00
Baidu | 100.00 | 1.01 (sd=0.10) | 99.00
Table 4. Performance of information collection. | Tutorials Match Function Description | Step Description
---|---|---
| Top-5 | Average | Top-1 | Accuracy
| Hit Rate(%) | Rank | Hit Rate(%) | (%)
Manual(Google) | 86.71 | 1.55 (sd=0.85) | 64.92 | -
Manual(Bing) | 89.17 | 1.36 (sd=0.80) | 69.38 | -
Manual(Baidu) | 97.21 | 1.42 (sd=2.76) | 87.96 | -
PromptRPA | 98.83 | 1.27 (sd=0.91) | 89.38 | 88.38
PromptRPA(AK+UI) | 100.00 | 1.22 (sd=0.74) | 90.83 | 99.21

However, prompts provided by users often lack complete accuracy, leading to a
decrease in effectiveness when using the original prompts for manual searches
on these websites. Using them as baselines, we tested the performance of
PromptRPA. As demonstrated in Table 4, our system improved the Top-5 hit rate
from 97.21% (Baidu) to 100%, while achieving a Top-1 hit rate of 90.83%. On
average, each prompt required 1.15 user interventions (sd = 0.52), which is
the number of conversations between the user and PromptRPA. Remarkably,
without user interventions, our system demonstrated a Top-1 hit rate of
89.38%, surpassing the baseline performance.

After the agent matched function descriptions with online tutorials, it
filtered concise step descriptions from complex web data. The accuracy of step
descriptions refers to the correct extraction from tutorial web pages,
ensuring they are free from fabrication, omissions, or errors. When the system
automatically selected the first candidate, the accuracy was 88.38%. When
users could select from five candidates, the accuracy rose to 99.21%.

####  5.3.2. Instruction Generation

Table 5. Performance of instruction generation. | Partial Accuracy(%) | Complete Accuracy(%)
---|---|---
Seq2act (Li et al., 2020a) | 70.13 | 54.00
PromptRPA | 84.55 | 73.00
PromptRPA(AK+UI) | 100.00 | 100.00

This stage evaluates the system’s ability to transform step descriptions into
a list of instructions in the predefined format.

Using the ground-truth step descriptions of the selected 100 RPA task models
as input, we evaluated correctness using two metrics, focusing solely on
textual interpretation:

  * •

Complete Accuracy: The percentage of RPA task models where all generated
instructions match the ground truth.

  * •

Partial Accuracy: The proportion of correct instructions relative to the total
ground-truth instructions.

We used the Seq2Act model (Li et al., 2020a), a transformer trained using the
ANDROIDHOWTO dataset, as our baseline (after translating step descriptions
into English). Table 5 reveals that PromptRPA achieved 100% accuracy for both
complete and partial metrics. On average, achieving perfection required 0.93
user interventions per RPA task (sd=1.34). It’s worth noting that while
perfection necessitated user interventions, the subsequent pipeline stages are
designed to accommodate minor errors, reducing the need for strict user
oversight. Without user intervention, the system attained 84.55% partial and
73.00% complete accuracy. In cold-start scenarios, PromptRPA has achieved
performance comparable to that achieved by related works (Li et al., 2020a; Vu
et al., 2023), which require substantial training data (at least 10K-level).

####  5.3.3. Operation Mapping

Table 6. Performance of operation mapping. | Prediction Accuracy(%) | Success Rate(%)
---|---|---
Text only | 61.49 | 44.00
PromptRPA | 81.66 | 76.00
PromptRPA(AK+UI) | 100.00 | 96.00

This stage evaluates the system’s ability to execute operations on
smartphones, guided by instructions. We used the ground-truth instruction
lists of the 100 RPA task models for testing.

Our baseline for comparison utilized strict text matching to identify
operation widgets. Table 6 shows the results. The _Grounding Agent_ ’s
prediction accuracy is measured by the ratio of operations that align with the
ground-truth operation sequences. With user intervention (i.e., users can
demonstrate any operations), the success rate reached 96.00%. The 4
unsuccessful RPA tasks were infeasible in the test environment. The
_Assessment Agent_ shows a 77.27% precision and a 97.06% recall rate for
identifying instances requiring user intervention. On average, each RPA task
required 0.68 user interventions (sd = 0.85). Even without user assistance,
the success rate was 76.00%, far surpassing the baseline.

Notably, not all instructions were executable on smartphones, usually due to
obsolescence from application updates. Our tests revealed that 81.33% of the
ground-truth instructions were feasible in the test environment. Thus, our
system does more than merely interpret the original instructions; it also
performs its own assessments and predictions based on encountered mobile
pages.

####  5.3.4. The Entire Pipeline

Table 7. Success rate (%) across various stages of PromptRPA. | Information Collection | Instruction Generation | Operation Mapping
---|---|---|---
GPT-4 only, Text only | - | 50.88 | 22.28
PromptRPA | 88.60 | 65.04 | 72.80
PromptRPA(AK) | 92.72 | 72.44 | 83.68
PromptRPA(AK+UI) | 99.12 | 99.12 | 95.21

To evaluate the overall effectiveness of our system, we input 2,500 textual
prompts into the pipeline, which comprises information collection, instruction
generation, and operation mapping. We did not use UGIF (Venkatesh et al.,
2022) as a baseline, because our tasks were not included in its tutorial
collection. We did not use Voicify (Vu et al., 2023) either, because its
direct invocation requires the apps’ code to contain deep links, but most of
our tasks did not satisfy it. Instead, we established our baseline using GPT-4
for direct instruction generation from user inputs, coupled with strict text
matching for operations.

Are shown in Table 7, the success rate for each stage is strictly determined
by the complete accuracy achieved up to that stage. Importantly, the overall
success rate is not a simple product of success rates at each stage, due to
the system’s built-in flexibility for error correction in downstream stages.
For instance, the operation mapping stage doesn’t strictly follow the
generated instructions; it also utilizes the agent’s explorative reasoning.
With user intervention, the average number of interventions per task is 1.66
(sd = 1.18). Notably, this number is smaller than the sum of interventions
required for each stage as reported earlier, which shows that the accumulation
of knowledge from completed tasks can reduce user interventions.

###  5.4. Error Analysis

We analyzed the errors that occurred in the three stages of PromptRPA.

####  5.4.1. Information Collection

Our system effectively matches tutorial web pages but falls short in content
extraction. Two main issues affect the performance. First, web pages
containing excessively long text compromise the LLM’s extraction capabilities,
leading to information loss and increased risk of generating inaccurate
content. Second, the LLM also tends to generate fabricated step descriptions
when interpreting the textual prompts.

To address these challenges, we have considered two options for improving text
quality. Initially, we considered implementing a specialized assessment agent,
similar to the one used in the operation mapping, to manage data quality.
However, we chose a human-in-the-loop approach, considering the efficiency of
manual text assessment and the extra time needed for an agent-based solution.
Our current system allows users to manually select high-quality step
descriptions for subsequent stages.

####  5.4.2. Instruction Generation

Figure 11. Errors of automatic instruction generation.

Figure 12. Errors of automatic operation mapping.

Our system’s automatic instruction generation achieves a partial accuracy of
84.55%, with errors distributed across six main categories as shown in Figure
12. These errors are generally resolved through simple user corrections.
Specifically, 43.75% of errors are due to instructions that go beyond the
existing definition (Table 4.1.3). Within this subset, 27.08% involve the LLM
using near-synonyms (e.g., “drag down” instead of “scroll down”). The
remaining 16.67% pertain to complex operations like “log in” or “register”
that the LLM struggles to parse, some of which can be resolved by the
exploration in the subsequent stage. Additionally, ‘Content fabrication’
represents the second largest category of errors, which is a notable issue
associated with the LLM. It occasionally does not strictly adhere to the given
step description and instead invents or elaborates on instructions.

Notably, not all errors necessitate user intervention. Many issues are
addressed in the subsequent stage, improving the overall success rate compared
to the instruction generation stage (Table 7). It is due to the system’s
ability to self-correct certain operations. Essentially, generated
instructions act as guidelines rather than absolutes, and even incorrect
instructions offer valuable context that enhances the system’s overall
performance.

####  5.4.3. Operation Mapping

The system achieves an 81.66% automatic prediction accuracy. As detailed in
Figure 12, errors can be categorized as follows:

Data Loading Failures: 8.24% are caused by issues in page data loading,
preventing the system from correctly identifying and operating widgets.

Semantic Ambiguity: 17.65% of errors arise from the system misidentifying
widgets due to semantic ambiguities and fuzzy matching challenges. In this
case, the agent can iterate attempts and refine solutions based on triggered
results.

Agent-Related Errors: 18.82% of errors are due to the _Grounding Agent_ taking
an unrecoverable erroneous direction in task execution. 14.12% of errors
happen when the _Assessment Agent_ wrongly rejects operations initially
predicted by the _Grounding Agent_. Such errors occur when contextual
information is insufficient, but they diminish as the _mobile interaction
graph_ expands. The graph can provide the agents with a richer context for
each operation’s implications.

Information Inadequacy: Errors due to incomplete page data constitute the
largest proportion, accounting for 41.18%. Widgets without textual
descriptions are particularly affected. To address this, the _Mobile Semantics
Agent_ employs advanced image-to-text models, although their ability to
distinguish finer details among mobile widgets is limited. Fortunately, users
can often provide context-specific interpretations to mitigate these issues.
For example, they might interpret the icons, or clarify that “select” in a
certain context means “switch” rather than “click”.

##  6\. User Evaluation

Given the influence of user feedback on system performance, we assessed
PromptRPA’s usability and robustness through engagement with diverse users.

###  6.1. Experiment Setup

Participant Demographics: We recruited a total of 12 participants for our
study. Their proficiency was determined through a pre-study questionnaire that
evaluated their comfort level with smartphone usage, experience with RPA
tools, and frequency of smartphone use. The participants were divided into the
following two categories according to their proficiency.

  * •

Skilled Users: 6 participants were proficient smartphone users (2 females;
aged from 18 to 40, M = 28.83), who were familiar with a wide range of
applications and functions.

  * •

Unskilled Users: 6 participants were less proficient in smartphone usage (4
females; aged from 50 to 69, M = 60.00). The group included elders, beginners,
or infrequent smartphone users.

App Selection: The study included tasks across 6 applications—3 commonly used
(WeChat, QQ, TikTok) and 3 less commonly used (Beautiful Weather, Safe and
Sound, Film Encyclopedia). The selection criteria for these applications are
consistent with those described in Section 5.1.

Task Selection: To ensure balanced task assignments, the participants were
grouped into 3 sets, each containing 4 participants (2 skilled and 2 unskilled
users). There are two types of tasks.

  * •

Predefined Tasks: The 4 participants within a set are assigned the same 8
predefined tasks, split across 2 applications: one commonly used and one less
commonly used. For each application, 2 common and 2 uncommon RPA tasks are
chosen, based on criteria detailed in Section 5.1. Part of the tasks are shown
in Table 2. In each four-participant set, we assume that ‘a’ and ‘b’ are
skilled users, whereas ‘c’ and ‘d’ are unskilled users. To ensure balance, for
each application, one common task is performed manually by ‘a’ and ‘c’, while
‘b’ and ‘d’ use PromptRPA. For the other common task, roles are reversed: ‘a’
and ‘c’ use PromptRPA, while ‘b’ and ‘d’ perform it manually. The approach is
similar for uncommon tasks.

  * •

User-Suggested Tasks: In addition to the predefined tasks, each participant is
allowed to propose 2 RPA tasks for each of their assigned applications. We
encouraged participants to suggest challenging tasks. For these 2 tasks,
participants will employ both manual execution and PromptRPA. To
counterbalance the effects of the method sequence, the order of methods is
alternated across tasks.

Methods: Almost all tasks in the experiment, requiring in-depth application
functionality, have been validated as unsupported by Google Assistant and
HUAWEI Celia, which excel in information gathering but fail to execute tasks.
For manual execution, participants are allowed to seek assistance from online
resources and people around them. For tasks carried out using PromptRPA,
participants can freely provide any information through textual prompts, which
can either describe the task’s goal or outline the detailed steps required for
the RPA task. After completing the RPA task, participants are required to use
a different expression to invoke the RPA task again.

Metrics: We recorded both the time and success rate for each task,
irrespective of whether it was performed manually or via PromptRPA. For tasks
carried out using PromptRPA, the frequency of user interventions was also
recorded.

Experimental Device: Participants used their own Android smartphones for the
experiment. PromptRPA’s client is installed on these devices.

In summary, we collected 192 ($12\times 8\times 2$) data points for tasks
completed using PromptRPA (half of which represent first-time usage and the
other half for second-time usage), and 96 ($12\times 8$) data points for tasks
completed manually.

###  6.2. Experimental Results

This section elucidates the experiment’s results, focusing on metrics
including success rate, completion time, and frequency of user interventions,
as well as subject feedback.

Table 8. Impact of different methods and participant groups on task success rate (%). | Manual | PromptRPA(First Time) | PromptRPA(Second Time)
---|---|---|---
Unskilled | 66.67 | 97.91 | 93.75
Skilled | 77.08 | 97.91 | 97.91

Figure 13. Impact of methods and participant groups on task completion time
(detailed in Table 9).

Figure 14. Impact of step count on task completion time.

Success Rate. Table 8 reveals that PromptRPA outperforms manual methods in
both skilled and unskilled groups in terms of task success rate. Failures
during the first use of PromptRPA were largely attributable to the low quality
of online tutorials for uncommonly used applications. Failures during the
second attempt were mainly due to page data loading issues that blocked access
to operation widgets.

Completion Time. Figure 14 shows that completing tasks with PromptRPA takes
significantly longer during the first use than when carried out manually
(p-value<0.001). The time stems primarily from the system’s retrieval for
online tutorials and exploration of on-screen operation sequences. Figure 14
indicates that the completion time correlates positively with the number of
steps involved during the first use of PromptRPA, which is more pronounced
than with manual methods and the second usage of PromptRPA. This latency is
tied to the efficiency limitations of the LLM. Notably, time decreases
substantially on second use, as the system’s historical RPA repository
eliminates the need for additional tutorial retrieval and exploration, thereby
becoming significantly faster than manual methods (p-value<0.001).

User Intervention. When using PromptRPA, each new task required 1.92 user
interventions (sd = 1.07) on average. Of these, 0.75 interventions per task
(sd = 1.04) occurred during the operation mapping. The _Assessment Agent_
accurately predicted the need for these interventions with a precision rate of
66.67% and a recall rate of 83.33%. Notably, the second time a task was
executed using PromptRPA, user interventions were no longer required.

Figure 15. Distribution of participant ratings for each statement in Table 10,
assessing various aspects of the system. Ratings range from 1 (strongly
disagree) to 7 (strongly agree).

Subjective Feedback. Post-experiment interviews were conducted with
participants, and their 7-point Likert scale responses are summarized in
Figure 15.

  * •

Low Effort: Participants generally found the system’s required effort
reasonable (5.42 in 7). They commonly noted that most interventions were for
locating small icons. P3 expressed surprise and satisfaction, _“Once I taught
it the process, next time I only needed to provide a few keywords to complete
the task.”_ However, P2 expressed a desire for more automation to reduce
mental effort.

  * •

Ease of Use: With a high average score of 6.33, participants found PromptRPA
user-friendly, helping them complete tasks they otherwise could not. P1
suggested incorporating images into the step descriptions for clarity, similar
to the online tutorials including screenshots. P4, an older user, remarked,
_“Adding steps is quite challenging for me; I hope the tutorials could be more
accurate.”_

  * •

Low Time Pressure: Participants generally understood the longer initial setup
time but expressed hopes for future improvements (5.83 in 7). P5 noted,
_“Although the process took a bit of time, the real-time feedback made it feel
like a learning experience, which was acceptably paced for me.”_

  * •

Decision and Interactive Intelligence: Users awarded high scores of 6.50 and
6.08, respectively, commending these aspects of the system. P3 specifically
praised the feature that _“aligns the smartphone’s operational page with each
step description”_. Similarly, P5 stated, _“Before, I struggled with the
tutorial’s first step not matching my current page. Now it is a significant
improvement.”_. However, room for improvement was noted. P9 advocated for a
more _“aggressive”_ execution approach, arguing the current system is slightly
conservative. P11 noted that some steps for confirmation were unnecessary. P12
observed, _“Tasks generally succeeded, but some took a roundabout path.”_

  * •

Interactive Flexibility: Participants felt unrestricted in expressing queries
(6.67 in 7). P1 found the context supplement very helpful, which allows for
queries without the need for _“careful consideration”_. Besides textual
expression, P3 expressed a desire for more intuitive communication methods,
_“Guess what I am trying to do and help me out when I am stuck in a certain
step.”_

  * •

Satisfaction and Willingness to Reuse: Overall, participants were very
satisfied with PromptRPA (6.25 in 7) and willing to reuse (6.58 in 7). Many
participants praised the system’s transparency and user-centered approach. P8
stated, _“It gives me a feeling like I’m navigating through the tasks. It’s
very interesting.”_ The utility of PromptRPA in simplifying otherwise complex
tasks was another strong point. P1 expressed, _“I often ask my family how to
use various features on my phone, but it’s challenging to follow their
instructions. Now I can just copy their responses and the system executes the
steps for me.”_ Similarly, P12 shared, _“I find new applications, especially
those with complex functions, very daunting. It’s such a hassle to contact
customer service and then follow their steps. PromptRPA saves me from all that
trouble.”_ P9 pointed out, _“I can define a few routine tasks, like accounting
after spending money. The agent can then automate these tasks for me.”_ P3
stated, _“I often forget the steps, so the replay is very useful for me.”_

##  7\. Discussion

In this section, we delve deeper into the significance and applications of our
proposed PromptRPA. We also discuss the limitations of the system and
potential future work.

###  7.1. Physical Path-finding v.s. Digital Navigation

We draw an analogy between physical path-finding and digital navigation to
elucidate PromptRPA’s implications. This comparison helps highlight the
universal principles embedded in PromptRPA, applicable in both physical and
digital realms.

Mapping Out Task Knowledge: Just as a physical journey begins with map
consultation, PromptRPA starts its digital journey with information
collection. By merging existing and new knowledge, it provides a detailed task
overview, similar to how a map provides guidance.

Plotting the Expedition: A traveler uses a map to chart a route, marking key
waypoints. Similarly, during instruction generation, our system transforms
complex data into concise, executable instructions.

Embarking on the Journey: Physically traveling a planned route parallels our
system’s operation mapping, where plans are not only devised but actualized,
dynamically adapting to the digital landscape’s ever-changing terrain.

In essence, PromptRPA introduces a general digital navigation framework. It
aims to convert information across modalities (from texts to operations), with
its stages focusing on bridging gaps between them and minimizing information
loss. Therefore, each stage can be further refined with domain-specific
enhancements, with its multi-agent framework providing great convenience.

###  7.2. Open and Accumulated Knowledge

In addition to the collaboration of its multi-agents, PromptRPA’s
effectiveness is derived from its open and accumulated knowledge, ensuring its
broad applicability and efficiency.

PromptRPA leverages massive online tutorials to address a broader range of
user needs, surpassing existing work that relies on limited datasets (Li et
al., 2020a; Venkatesh et al., 2022), and eliminating the need for prior data
collection (Vu et al., 2023). It supports functionalities deep within
applications, which exceeds the capabilities of current assistants like Google
Assistant and HUAWEI Celia. Moreover, PromptRPA’s approach to mobile semantics
is open-ended, enabling exploration beyond rigid adherence to given
instructions. The _Mobile Semantics Agent_ interprets real-time page contexts,
and in conjunction with existing instructions, the _Grounding Agent_ predicts
the operation widgets. The open-ended approach allows PromptRPA to handle
diverse textual inputs and adapt to changing environments, significantly
expanding RPA’s application scope.

PromptRPA’s knowledge base evolves through user usage and feedback. The
evolution manifests in improved success rates, rising from 72.80% to 95.21%,
and reduced completion time for repeated tasks (Figure 14). Notably, the
accumulation of knowledge contributes to a reduction in user interventions.
When similar task requirements arise again, PromptRPA operates autonomously,
only requiring user intervention for contextual parameter adjustments or
significant interface updates. The accumulated knowledge empowers PromptRPA to
tackle increasingly complex and nuanced tasks, dynamically adapting to
emerging user needs.

###  7.3. Applications

By incorporating a vast array of external knowledge and accepting various
textual inputs, PromptRPA supports a wide range of task executions, benefiting
both end users and developers.

For end users, PromptRPA improves the usability of smartphones and facilitates
assistance between peers. It offers guided support for users navigating
unfamiliar smartphone functions, enabling them to access and automate tasks
deep within applications. Its capacity to handle a broader range of tasks
allows users to personalize their smartphone experience by automating their
own complex or repetitive tasks. Furthermore, textual assistance from various
sources, such as family, friends, or online customer service, can be fed to
PromptRPA for automating tasks, which is valuable for teamwork or external
support. For instance, an elderly user receiving textual assistance from
family members can simply input the text into PromptRPA for automating tasks,
eliminating the need for constant reference to instructions and navigating
through complex interfaces.

For developers, PromptRPA can reduce the costs of developing and maintaining
task automation. It can cost-effectively transform a vast array of existing
tutorial texts into operations, seamlessly integrated into current voice
assistants or online customer service, thereby expanding the range of
supported tasks. Application and platform developers can leverage PromptRPA to
transform text-based cases in bulk into interactive tutorials or executable
test scripts. It significantly reduces the cost of updating voice assistants,
interactive tutorials, and test scripts, as developers need only update the
source texts regularly to reflect the latest application functions and
interfaces. Nonetheless, in sensitive applications such as banking,
implementing robust authentication mechanisms is crucial to prevent malicious
command execution.

###  7.4. Limitations and Future Work

We outline the limitations of PromptRPA, along with possible directions for
future enhancements.

Adapting Dialogue Mechanisms for Diverse User Groups: The current design of
the _Analysis Agent_ offers a generalized approach without catering to the
specific needs of different user demographics. For instance, older and younger
users may articulate issues differently, necessitating a more nuanced,
tailored dialogue system to accommodate their needs.

Parsing Complex Tutorial Logic: While our system can parse nested tutorial
texts, there is room for improvement. For example, automating responses to
conditional branches like “If the version number is less than 14.0, please
upgrade the app” could enhance the system’s efficiency. Future work will focus
on more sophisticated logic analysis, including nested conditions.

Improving Semantic Understanding of User Interfaces: In the present design,
the _Mobile Semantics Agent_ emphasizes the textual descriptions of mobile
pages. However, current models often fall short, particularly when faced with
pages not meeting accessibility standards. It results in a notable loss of
information during the translation from interface visuals to text. Moving
forward, we aim to enhance algorithms that better recognize semantic
structures within GUI, with the potential improvement of image recognition
technologies.

Balancing Comprehensive Coverage with In-Depth Improvement: Currently, in
order to expand the application scope, PromptRPA applies the capabilities of
LLM across various stages. However, it has not deeply explored some domains,
such as addressing ambiguities in natural language and setting instruction
parsing templates to reduce the risk of LLM fabrication. As PromptRPA’s usage
data increases, we will systematically collect domain-specific data and user
feedback to fine-tune the LLM or train local models, gradually reducing
reliance on generalized templates and increasing its capacity for nuanced,
context-aware processing.

##  8\. Conclusion

This paper presents PromptRPA as an effective solution in Robotic Process
Automation, particularly addressing the barriers of technical complexity that
deter its mass adoption. Utilizing the multi-agent framework and supplementing
online tutorials, PromptRPA is capable of interpreting unrestricted textual
prompts to perform a wide range of RPA tasks. Performance evaluations showcase
a marked rise in success rates, ascending from an initial 22.28% to a
remarkable 95.21%, with an average of only 1.66 user interventions per new
task. Even without user interventions, a success rate of 83.68% is achieved.
User studies further validate the effectiveness of PromptRPA in delivering a
user-friendly and intelligent experience. Beyond demonstrating the viability
of text-driven control in RPA, our research paves the way for continued
exploration, emphasizing enhanced functionality and versatility. Upcoming
efforts aim to minimize the burden of user interaction, engage a wider user
group, and boost the system’s proficiency in managing complex tasks.

## References

  * (1)
  * Aguirre and Rodriguez (2017) Santiago Aguirre and Alejandro Rodriguez. 2017.  Automation of a Business Process Using Robotic Process Automation (RPA): A Case Study. In _Applied Computer Sciences in Engineering_ , Juan Carlos Figueroa-García, Eduyn Ramiro López-Santana, José Luis Villa-Ramírez, and Roberto Ferro-Escobar (Eds.). Springer International Publishing, Cham, 65–71.
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. 2022.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.  arXiv:2204.01691 [cs.RO]
  * Amershi et al. (2014) Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014.  Power to the People: The Role of Humans in Interactive Machine Learning.  _AI Magazine_ 35, 4 (Dec. 2014), 105–120.  https://doi.org/10.1609/aimag.v35i4.2513 Number: 4.
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Beneteau et al. (2019) Erin Beneteau, Olivia K. Richards, Mingrui Zhang, Julie A. Kientz, Jason Yip, and Alexis Hiniker. 2019.  Communication Breakdowns Between Families and Alexa. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300473
  * Berg et al. (2019) Stuart Berg, Dominik Kutra, Thorben Kroeger, Christoph N Straehle, Bernhard X Kausler, Carsten Haubold, Martin Schiegg, Janez Ales, Thorsten Beier, Markus Rudy, et al. 2019\.  Ilastik: interactive machine learning for (bio) image analysis.  _Nature methods_ 16, 12 (2019), 1226–1232.
  * Branavan et al. (2009) S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009.  Reinforcement Learning for Mapping Instructions to Actions. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_. Association for Computational Linguistics, Suntec, Singapore, 82–90.  https://aclanthology.org/P09-1010
  * Burns et al. (2022) Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022.  A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility.  http://arxiv.org/abs/2202.02312 arXiv:2202.02312 [cs].
  * Carney et al. (2020) Michelle Carney, Barron Webster, Irene Alvarado, Kyle Phillips, Noura Howell, Jordan Griffith, Jonas Jongejan, Amit Pitaru, and Alexander Chen. 2020.  Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI EA ’20)_. Association for Computing Machinery, New York, NY, USA, 1–8.  https://doi.org/10.1145/3334480.3382839
  * Castelvecchi (2016) Davide Castelvecchi. 2016.  Can we open the black box of AI?  _Nature News_ 538, 7623 (2016), 20.
  * Chakraborti et al. (2020) Tathagata Chakraborti, Vatche Isahagian, Rania Khalaf, Yasaman Khazaeni, Vinod Muthusamy, Yara Rizk, and Merve Unuvar. 2020.  From Robotic Process Automation to Intelligent Process Automation. In _Business Process Management: Blockchain and Robotic Process Automation Forum_ , Aleksandre Asatiani, José María García, Nina Helander, Andrés Jiménez-Ramírez, Agnes Koschmider, Jan Mendling, Giovanni Meroni, and Hajo A. Reijers (Eds.). Springer International Publishing, Cham, 215–228.
  * Chen et al. (2022a) Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022a.  VisualGPT: Data-Efficient Adaptation of Pretrained Language Models for Image Captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 18030–18040.
  * Chen et al. (2022b) Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. 2022b.  Towards Complete Icon Labeling in Mobile Applications. In _CHI Conference on Human Factors in Computing Systems_. ACM, New Orleans LA USA, 1–14.  https://doi.org/10.1145/3491102.3502073
  * Chen et al. (2017) Xiaoou Chen, Fei Wang, Zhenwei You, Xiaochun Wang, Chunjing Tao, and Jian Liu. 2017.  Design of Interactive Tutorials on Mobile Applications for Chinese Middle-Aged and Older Adults.  _Art and Design Review_ 05, 03 (2017), 162.  https://doi.org/10.4236/adr.2017.53013 Number: 03 Publisher: Scientific Research Publishing.
  * de Medeiros et al. (2020) André Dantas de Medeiros, Nayara Pereira Capobiango, Jose Maria da Silva, Laercio Junio da Silva, Clissia Barboza da Silva, and Denise Cunha Fernandes dos Santos Dias. 2020.  Interactive machine learning for soybean seed and seedling quality classification.  _Scientific reports_ 10, 1 (2020), 11267.
  * Deka et al. (2017) Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_. ACM, Québec City QC Canada, 845–854.  https://doi.org/10.1145/3126594.3126651
  * Deng et al. (2020) Changyu Deng, Xunbi Ji, Colton Rainey, Jianyu Zhang, and Wei Lu. 2020.  Integrating Machine Learning with Human Knowledge.  _iScience_ 23, 11 (Nov. 2020), 101656.  https://doi.org/10.1016/j.isci.2020.101656
  * Dixon et al. (2014) Morgan Dixon, Gierad Laput, and James Fogarty. 2014.  Pixel-based methods for widget state and style in a runtime implementation of sliding widgets. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Toronto, Ontario, Canada) _(CHI ’14)_. Association for Computing Machinery, New York, NY, USA, 2231–2240.  https://doi.org/10.1145/2556288.2556979
  * Dorri et al. (2018) Ali Dorri, Salil S. Kanhere, and Raja Jurdak. 2018.  Multi-Agent Systems: A Survey.  _IEEE Access_ 6 (2018), 28573–28593.  https://doi.org/10.1109/ACCESS.2018.2831228
  * Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023.  Improving Factuality and Reasoning in Language Models through Multiagent Debate.  arXiv:2305.14325 [cs.CL]
  * Dudley and Kristensson (2018) John J. Dudley and Per Ola Kristensson. 2018.  A Review of User Interface Design for Interactive Machine Learning.  _ACM Trans. Interact. Intell. Syst._ 8, 2, Article 8 (jun 2018), 37 pages.  https://doi.org/10.1145/3185517
  * Fails and Olsen (2003) Jerry Alan Fails and Dan R. Olsen. 2003.  Interactive Machine Learning. In _Proceedings of the 8th International Conference on Intelligent User Interfaces_ (Miami, Florida, USA) _(IUI ’03)_. Association for Computing Machinery, New York, NY, USA, 39–45.  https://doi.org/10.1145/604045.604056
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic BERT Sentence Embedding.  arXiv:2007.01852 [cs.CL]
  * Fraser et al. (2019) C. Ailie Fraser, Tricia J. Ngoon, Mira Dontcheva, and Scott Klemmer. 2019.  RePlay: Contextually Presenting Learning Videos Across Software Applications. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300527
  * Grudin and Jacques (2019) Jonathan Grudin and Richard Jacques. 2019.  Chatbots, Humbots, and the Quest for Artificial General Intelligence. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–11.  https://doi.org/10.1145/3290605.3300439
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018.  A survey of methods for explaining black box models.  _ACM computing surveys (CSUR)_ 51, 5 (2018), 1–42.
  * He et al. (2021) Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, and Jindong Chen. 2021.  ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 7 (May 2021), 5931–5938.  https://doi.org/10.1609/aaai.v35i7.16741 Number: 7.
  * Herm et al. (2023) Lukas-Valentin Herm, Christian Janiesch, Alexander Helm, Florian Imgrund, Adrian Hofmann, and Axel Winkelmann. 2023.  A framework for implementing robotic process automation projects.  _Information Systems and e-Business Management_ 21, 1 (2023), 1–35.
  * Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. 2023.  MetaGPT: Meta Programming for Multi-Agent Collaborative Framework.  arXiv:2308.00352 [cs.AI]
  * Huang et al. (2023) Tian Huang, Chun Yu, Weinan Shi, Bowen Wang, David Yang, Yihao Zhu, Zhaoheng Li, and Yuanchun Shi. 2023.  Interaction Proxy Manager: Semantic Model Generation and Run-Time Support for Reconstructing Ubiquitous User Interfaces of Mobile Services.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 7, 3, Article 99 (sep 2023), 39 pages.  https://doi.org/10.1145/3610929
  * Islam et al. (2010) Rashedul Islam, Rofiqul Islam, and Tohidul Mazumder. 2010.  Mobile application and its global impact.  _International Journal of Engineering & Technology_ 10, 6 (2010), 72–78.
  * Jansen et al. (1998) Bernard J. Jansen, Amanda Spink, Judy Bateman, and Tefko Saracevic. 1998.  Real life information retrieval: a study of user queries on the Web.  _ACM SIGIR Forum_ 32, 1 (April 1998), 5–17.  https://doi.org/10.1145/281250.281253
  * Jin et al. (2022) Xiaofu Jin, Xiaozhu Hu, Xiaoying Wei, and Mingming Fan. 2022.  Synapse: Interactive Guidance by Demonstration with Trial-and-Error Support for Older Adults to Use Smartphone Apps.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 6, 3, Article 121 (sep 2022), 24 pages.  https://doi.org/10.1145/3550321
  * Jovanović et al. (2018) Stefan Z Jovanović, Jelena S Đurić, and Tatjana V Šibalija. 2018.  Robotic process automation: overview and opportunities.  _International Journal Advanced Quality_ 46, 3-4 (2018), 34–39.
  * Lee et al. (2022) Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2022.  Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding.  http://arxiv.org/abs/2210.03347 arXiv:2210.03347 [cs].
  * Lee et al. (2023) Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023.  Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding. In _Proceedings of the 40th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 202)_ , Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 18893–18912.  https://proceedings.mlr.press/v202/lee23g.html
  * Li et al. (2015) Guo Li, Tun Lu, Jiang Yang, Xiaomu Zhou, Xianghua Ding, and Ning Gu. 2015.  Intelligently Creating Contextual Tutorials for GUI Applications. In _2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom)_. IEEE, Beijing, 187–196.  https://doi.org/10.1109/UIC-ATC-ScalCom-CBDCom-IoP.2015.50
  * Li et al. (2017) Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017.  SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ (Denver, Colorado, USA) _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 6038–6049.  https://doi.org/10.1145/3025453.3025483
  * Li et al. (2018) Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers. 2018.  APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions. In _2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. 105–114.  https://doi.org/10.1109/VLHCC.2018.8506506 ISSN: 1943-6106.
  * Li et al. (2021) Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021.  Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. ACM, Yokohama Japan, 1–15.  https://doi.org/10.1145/3411764.3445049
  * Li and Riva (2018) Toby Jia-Jun Li and Oriana Riva. 2018.  Kite: Building Conversational Bots from Mobile Apps. In _Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services_ (Munich, Germany) _(MobiSys ’18)_. Association for Computing Machinery, New York, NY, USA, 96–109.  https://doi.org/10.1145/3210240.3210339
  * Li (2021) Wei Li. 2021.  Learning UI Navigation through Demonstrations composed of Macro Actions.  http://arxiv.org/abs/2110.08653 arXiv:2110.08653 [cs].
  * Li et al. (2020a) Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020a.  Mapping Natural Language Instructions to Mobile UI Action Sequences. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 8198–8210.  https://doi.org/10.18653/v1/2020.acl-main.729
  * Li et al. (2020b) Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020b.  Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements.  http://arxiv.org/abs/2010.04295 arXiv:2010.04295 [cs].
  * Liang et al. (2023) Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023.  Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate.  arXiv:2305.19118 [cs.CL]
  * Liu et al. (2018) Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018.  Learning Design Semantics for Mobile Apps. In _Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology_ (¡conf-loc¿, ¡city¿Berlin¡/city¿, ¡country¿Germany¡/country¿, ¡/conf-loc¿) _(UIST ’18)_. Association for Computing Machinery, New York, NY, USA, 569–579.  https://doi.org/10.1145/3242587.3242650
  * Ni et al. (2020) Pin Ni, Yuming Li, Gangmin Li, and Victor Chang. 2020.  Natural language understanding approaches based on joint task of intent detection and slot filling for IoT voice interaction.  _Neural Computing and Applications_ 32 (2020), 16149–16166.
  * Palo et al. (2023) Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller. 2023.  Towards A Unified Agent with Foundation Models.  arXiv:2307.09668 [cs.RO]
  * Pan et al. (2022) Lihang Pan, Chun Yu, JiaHui Li, Tian Huang, Xiaojun Bi, and Yuanchun Shi. 2022.  Automatically Generating and Improving Voice Command Interface from Operation Sequences on Smartphones. In _CHI Conference on Human Factors in Computing Systems_. ACM, New Orleans LA USA, 1–21.  https://doi.org/10.1145/3491102.3517459
  * Panait and Luke (2005) Liviu Panait and Sean Luke. 2005.  Cooperative multi-agent learning: The state of the art.  _Autonomous agents and multi-agent systems_ 11 (2005), 387–434.  https://doi.org/10.1007/s10458-005-2631-2
  * Passini et al. (2008) Stefano Passini, Filiberto Strazzari, and Annamaria Borghi. 2008.  Icon-function relationship in toolbar icons.  _Displays_ 29, 5 (2008), 521–525.  https://doi.org/10.1016/j.displa.2008.07.001
  * Pasupat et al. (2018) Panupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, and Percy Liang. 2018.  Mapping Natural Language Commands to Web Elements.  arXiv:1808.09132 [cs.CL]
  * Pongnumkul et al. (2011) Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev, Shai Avidan, and Michael F. Cohen. 2011.  Pause-and-Play: Automatically Linking Screencast Video Tutorials with Applications. In _Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology_ (Santa Barbara, California, USA) _(UIST ’11)_. Association for Computing Machinery, New York, NY, USA, 135–144.  https://doi.org/10.1145/2047196.2047213
  * Rawles et al. (2023) Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023.  Android in the Wild: A Large-Scale Dataset for Android Device Control.  arXiv:2307.10088 [cs.LG]
  * Ribeiro et al. (2021) Jorge Ribeiro, Rui Lima, Tiago Eckhardt, and Sara Paiva. 2021.  Robotic Process Automation and Artificial Intelligence in Industry 4.0 – A Literature review.  _Procedia Computer Science_ 181 (2021), 51–58.  https://doi.org/10.1016/j.procs.2021.01.104 CENTERIS 2020 - International Conference on ENTERprise Information Systems / ProjMAN 2020 - International Conference on Project MANagement / HCist 2020 - International Conference on Health and Social Care Information Systems and Technologies 2020, CENTERIS/ProjMAN/HCist 2020.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language Models Can Teach Themselves to Use Tools.  arXiv:2302.04761 [cs.CL]
  * Shen et al. (2023) Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith Hentel, Beatriu Reig, George Shih, and Linda Moy. 2023.  ChatGPT and Other Large Language Models Are Double-edged Swords.  _Radiology_ 307 (01 2023).  https://doi.org/10.1148/radiol.230163
  * Singh et al. (2023) Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023.  ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. 11523–11530.  https://doi.org/10.1109/ICRA48891.2023.10161317
  * Smith et al. (2006) Benjamin Thomas Smith, Sergey Brin, Sanjay Ghemawat, and John Abraham Bauer. 2006.  Methods and apparatus for providing search results in response to an ambiguous search query.  US Patent 7,136,854.
  * Sperrle et al. (2019) Fabian Sperrle, Rita Sevastjanova, Rebecca Kehlbeck, and Mennatallah El-Assady. 2019.  VIANA: Visual Interactive Annotation of Argumentation. In _2019 IEEE Conference on Visual Analytics Science and Technology (VAST)_. 11–22.  https://doi.org/10.1109/VAST47406.2019.8986917
  * Van der Aalst et al. (2018) Wil MP Van der Aalst, Martin Bichler, and Armin Heinzl. 2018.  Robotic process automation.  , 269–272 pages.  https://doi.org/10.1007/s12599-018-0542-4
  * Venkatesh et al. (2022) Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. 2022.  UGIF: UI Grounded Instruction Following.  http://arxiv.org/abs/2211.07615 arXiv:2211.07615 [cs].
  * Vu et al. (2023) Minh Duc Vu, Han Wang, Zhuang Li, Gholamreza Haffari, Zhenchang Xing, and Chunyang Chen. 2023.  Voicify Your UI: Towards Android App Control with Voice Commands.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 7, 1, Article 44 (mar 2023), 22 pages.  https://doi.org/10.1145/3581998
  * Wang et al. (2023) Bryan Wang, Gang Li, and Yang Li. 2023.  Enabling Conversational Interaction with Mobile UI using Large Language Models.  http://arxiv.org/abs/2209.08655 arXiv:2209.08655 [cs].
  * Wang et al. (2021) Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021.  Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. In _The 34th Annual ACM Symposium on User Interface Software and Technology_. ACM, Virtual Event USA, 498–510.  https://doi.org/10.1145/3472749.3474765
  * Wang et al. (2014) Cheng-Yao Wang, Wei-Chen Chu, Hou-Ren Chen, Chun-Yen Hsu, and Mike Y. Chen. 2014.  EverTutor: automatically creating interactive guided tutorials on smartphones by user demonstration. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, Toronto Ontario Canada, 4027–4036.  https://doi.org/10.1145/2556288.2557407
  * Wei et al. (2023) Jimmy Wei, Kurt Shuster, Arthur Szlam, Jason Weston, Jack Urbanek, and Mojtaba Komeili. 2023.  Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models.  arXiv:2304.13835 [cs.CL]
  * Wen et al. (2001) Ji-Rong Wen, Jian-Yun Nie, and Hong-Jiang Zhang. 2001.  Clustering User Queries of a Search Engine. In _Proceedings of the 10th International Conference on World Wide Web_ (Hong Kong, Hong Kong) _(WWW ’01)_. Association for Computing Machinery, New York, NY, USA, 162–168.  https://doi.org/10.1145/371920.371974
  * Wewerka et al. (2020) Judith Wewerka, Sebastian Dax, and Manfred Reichert. 2020.  A User Acceptance Model for Robotic Process Automation. In _2020 IEEE 24th International Enterprise Distributed Object Computing Conference (EDOC)_. 97–106.  https://doi.org/10.1109/EDOC49727.2020.00021
  * White et al. (2019) Thomas D. White, Gordon Fraser, and Guy J. Brown. 2019.  Improving random GUI testing with image-based widget detection. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Beijing, China) _(ISSTA 2019)_. Association for Computing Machinery, New York, NY, USA, 307–317.  https://doi.org/10.1145/3293882.3330551
  * Winograd (1972) Terry Winograd. 1972.  Understanding natural language.  _Cognitive Psychology_ 3, 1 (1972), 1–191.  https://doi.org/10.1016/0010-0285(72)90002-3
  * Wu et al. (2023) Jason Wu, Rebecca Krosnick, Eldon Schoop, Amanda Swearngin, Jeffrey P Bigham, and Jeffrey Nichols. 2023.  Never-Ending Learning of User Interfaces. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (San Francisco, CA, USA) _(UIST ’23)_. Association for Computing Machinery, New York, NY, USA, Article 113, 13 pages.  https://doi.org/10.1145/3586183.3606824
  * Xie et al. (2020) Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020.  UIED: a hybrid tool for GUI element detection. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 1655–1659.  https://doi.org/10.1145/3368089.3417940
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  ReAct: Synergizing Reasoning and Acting in Language Models.  arXiv:2210.03629 [cs.CL]
  * Zhang et al. (2023) Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. 2023.  Building Cooperative Embodied Agents Modularly with Large Language Models.  arXiv:2307.02485 [cs.AI]
  * Zhao et al. (2023) Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2023.  ExpeL: LLM Agents Are Experiential Learners.  arXiv:2308.10144 [cs.LG]
  * Zhong et al. (2021) Mingyuan Zhong, Gang Li, Peggy Chi, and Yang Li. 2021.  HelpViz: Automatic Generation of Contextual Visual Mobile Tutorials from Text-Based Instructions. In _The 34th Annual ACM Symposium on User Interface Software and Technology_ _(UIST ’21)_. Association for Computing Machinery, New York, NY, USA, 1144–1153.  https://doi.org/10.1145/3472749.3474812

##  Appendix A Mult-Agent Design

###  A.1. Grounding Agent

In this section, we provide detailed descriptions of the strategies employed
by the _Grounding Agent_. Initially, based on the current page’s layout
hierarchy, the agent identifies and selects all widgets that either possess
text or have interactive attributes, thereby constructing a set of potential
operation candidates. Subsequently, the agent employs the following strategies
to predict the operational widget within these candidates:

  1. (1)

Historical RPA invocation: If the instruction has been previously mapped to an
operation, the agent has the capability to refer to stored widget data in the
historical RPA repository. It enables the agent to direct the operation to a
corresponding widget on the current interface. To evaluate the similarity
between the candidate widget and the stored widget, we employ a heuristic
rule-based similarity function, as described in the work of Huang et al.
(Huang et al., 2023). This function considers 26 features of a widget,
including its style and position in images, its style and position in layouts,
and its role in page segmentation. The similarity thresholds have been
established using Receiver Operating Characteristic (ROC) curve analysis.

  2. (2)

New instruction-based grounding: For new instructions, the agent then employs
the following strategies:

    1. (a)

Strict text matching: It involves searching for a widget on the page that
closely matches the object description ($Instruction.Object$). Similarity is
defined as Equation 1, where $Widget.text$ denotes the text and content
description contained in the widget’s layout information, and $LCS$ signifies
the length of the longest common sub-string between two strings. A match is
confirmed only if the similarity exceeds an empirically preset threshold.

(1) $\displaystyle Sim(Widget,Instruction.Object)$
$\displaystyle=\frac{2\times
LCS(Widget.text,Instruction.Object.text)}{length(Widget.text)+length(Instruction.Object.text)}$

    2. (b)

Textual semantic matching: This approach leverages the operation and object
description from the instruction. The agent enumerates potential widgets with
relevant information, including their text, content description, position on
the current page, and interactive attributes such as clickability and
editability. Subsequently, the LLM considers each widget’s semantic relevance
to the instruction and then predicts the most suitable widget for executing
the given instruction.

    3. (c)

Exploratory mechanism: In instances where the operation widget proves
difficult to identify, this mechanism becomes particularly instrumental. The
agent presents the LLM with a wealth of contextual information to refer to:

      1. (i)

The function description in the model.

      2. (ii)

The system’s current reference instructions, upcoming potential instructions,
and completed instructions.

      3. (iii)

Detailed attributes of all interactive widgets, such as their text, position
on the current page, and interactive attributes.

      4. (iv)

Descriptions of future pages each interactive widget may lead to. If these
pages’ layout and screenshot data have been stored in the _mobile interaction
graph_ , the _Mobile Semantics Agent_ can generate textual descriptions for
them.

      5. (v)

The reason for the previous prediction that the agent responded.

      6. (vi)

The evaluation made by the _Assessment Agent_ on the previous prediction.

###  A.2. Assessment Agent

The _Assessment Agent_ utilizes the LLM with a rich set of contextual
information, encompassing:

  * •

Function Description: It outlines the context or purpose of the current task.

  * •

Current Page Description: Sourced from the _Mobile Semantics Agent_ , it
provides context about the mobile GUI that the agent is interacting with.

  * •

Operation History: A log of operations that the agent has previously
performed, useful for understanding the context and making future predictions.

  * •

Current and Future Instructions: The immediate instruction the agent is
focusing on, as well as potential future instructions that might influence the
current operation. Although these instructions may not be strictly adhered to,
they serve as valuable reference points.

  * •

Details about the Predicted Operation: In addition to information about the
widgets predicted by the _Grounding Agent_ , it also incorporates reasoning
information from the _Grounding Agent_ :

    * –

The rationale behind the _Grounding Agent_ ’s prediction, explaining why this
operation was chosen.

    * –

Potential effects of the predicted operation, such as future pages the
operation may lead to, which might be found in the _mobile interaction graph_
, to facilitate reasoned decisions.

  * •

History of Attempts: The agent keeps a record of previously attempted and
failed interactions with pages and widgets. This is designed to prevent dead
loops during the exploration process.

The LLM is then tasked with choosing between the current predicted operation
and a possible rollback, also furnishing a confidence score for its decision.
The confidence score is used to determine whether human intervention is
necessary. An empirical threshold is set, below which the system automatically
prompts the user for confirmation.

The agent has several built-in guidelines for pathway exploration, to
effectively oversee the decision-making of the _Grounding Agent_ :

  * •

Limit Scrolling: The agent minimizes excessive scrolling on a page to avoid
worthless exploration.

  * •

Rationality Check: When considering the addition of new steps, such as
clicking a widget to explore tasks, the agent evaluates their logical
solidity.

  * •

Outcome Validation: Should the outcome of a previous step fail to meet
expectations or prove irrelevant to the RPA task, the agent opts for the
‘RETRACT’ option.

  * •

Task Completion: If the agent determines that the task is complete, it selects
the ‘FINISH’ option, discarding unnecessary steps.

##  Appendix B Accumulated Knowledge

###  B.1. Construction of the Historical RPA Repository

The Historical RPA Repository serves as an archive for successfully executed
RPA tasks. Since the _Grounding Agent_ doesn’t strictly adhere to pre-
generated instructions, the repository focuses on storing the actual sequences
of operations executed. This subsection details how these operation sequences
are reverse-engineered into corresponding instructions and step descriptions.

  1. (1)

Operation Sequence Storage: The first step is to archive the actual sequence
of operations executed by the _Grounding Agent_ , which is the most reliable
record of the RPA task. The system archives all layout hierarchical
information and screenshots of the pages visited. It also logs the operation
object’s index in the page’s layout tree to retrieve the object’s layout
information. Additionally, since the object’s position is recorded, it allows
for capturing its corresponding image on the full-page screenshot.

  2. (2)

Reverse Engineering to Instructions: Using the stored operation sequences, the
system reverse-engineers to produce the corresponding instructions. Each
operation is traced back to the potential instruction that could have prompted
it, adhering to the format specified in Table 4.1.3. In this context, the
operation types and parameters have already been logged by the system during
the triggering of the operation. The descriptions of operation objects are
derived from their texts, content descriptions, positions, and the average RGB
values of their respective images.

  3. (3)

Generation of Step Descriptions: Alongside instructions, the system also
formulates step descriptions. They are rephrased versions of the instructions,
articulated in natural language. While these historical step descriptions may
not directly contribute to the system’s operational efficiency, they enhance
the user experience by making the automated processes more transparent and
easier to understand.

###  B.2. Construction of the Context Library

The context library plays a crucial role in enhancing the system’s capacity to
interpret vague textual prompts by providing an array of mobile-specific
parameters. Here is the method employed for constructing the library.

  1. (1)

Online Q&A Data Collection: Initially, we sourced a set of 20,265 questions
related to mobile phone usage from a recognized online Q&A
platform555https://zhidao.baidu.com/.

  2. (2)

Participant Involvement: The collected questions were distributed among 24
undergraduate college students who participated in a formative study.

  3. (3)

Contextual Parameter Extraction: Each participant was tasked with identifying
the “central” keywords in each question. They then examined various variations
of the question to infer the contextual parameters required for comprehensive
understanding.

  4. (4)

Context Library Formation: We employed experts to review, analyze, and
categorize the list of contextual parameters identified by the participants.
Their insights were integrated to develop the finalized context library.

##  Appendix C User Evaluation

###  C.1. Objective Results

The impact of methods and participant groups on task completion time is shown
in Table 9.

Table 9. Impact of different methods and participant groups on task completion time (in seconds). | Manual | First Time Using PromptRPA | Second Time Using PromptRPA
---|---|---|---
Unskilled | 110.47 (sd = 54.86) | 249.45 (sd = 84.47) | 33.11 (sd = 6.14)
Skilled | 88.86 (sd = 55.51) | 212.09 (sd = 83.66) | 32.49 (sd = 7.83)

###  C.2. Subject Feedback

After the participants completed the experiment, we conducted interviews with
them. The results of the 7-point Likert scale are shown in Table 10.

Table 10. Subjective feedback. Statements | Results
---|---
Providing feedback to the system is effortless. | 5.42 (sd=1.16)
You can finish the task easily. | 6.33 (sd=0.65)
You find the time taken to be acceptable. | 5.83 (sd=1.47)
The system’s decisions are sufficiently intelligent. | 6.50 (sd=0.52)
The system’s requests for user assistance are appropriately made. | 6.08 (sd=0.51)
You can feel free to query. | 6.67 (sd=0.65)
The experience is good. | 6.25 (sd=1.06)
You are willing to use our PromptRPA. | 6.58 (sd=0.67)