# Towards Collaborative Plan Acquisition
through Theory of Mind Modeling in Situated Dialogue

Cristian-Paul Bara1, Ziqiao Ma1, Yingzhuo Yu1, Julie Shah2, Joyce Chai1
1University of Michigan, 2Massachussets Institute of Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Collaborative tasks often begin with partial task knowledge and incomplete
initial plans from each partner. To complete these tasks, agents need to
engage in situated communication with their partners and coordinate their
partial plans towards a complete plan to achieve a joint task goal. While such
collaboration seems effortless in a human-human team, it is highly challenging
for human-AI collaboration. To address this limitation, this paper takes a
step towards collaborative plan acquisition, where humans and agents strive to
learn and communicate with each other to acquire a complete plan for joint
tasks. Specifically, we formulate a novel problem for agents to predict the
missing task knowledge for themselves and for their partners based on rich
perceptual and dialogue history. We extend a situated dialogue benchmark for
symmetric collaborative tasks in a 3D blocks world and investigate
computational strategies for plan acquisition. Our empirical results suggest
that predicting the partner’s missing knowledge is a more viable approach than
predicting one’s own. We show that explicit modeling of the partner’s dialogue
moves and mental states produces improved and more stable results than
without. These results provide insight for future AI agents that can predict
what knowledge their partner is missing and, therefore, can proactively
communicate such information to help their partner acquire such missing
knowledge toward a common understanding of joint tasks.

##  1 Introduction

With the simultaneous progress in AI assistants and robotics, it is reasonable
to anticipate the forthcoming milestone of embodied assistants. However, a
critical question arises: how can we facilitate interactions between humans
and robots to be as intuitive and seamless as possible? To bridge this gap, a
significant challenge lies in the mismatched knowledge and skills between
humans and agents, as well as their tendency to begin with incomplete and
divergent partial plans. Considering that the average user cannot be expected
to possess expertise in robotics, language communications become paramount.
Consequently, it becomes imperative for humans and agents to engage in
effective language communication to establish shared plans for collaborative
tasks. While such coordination and communication occur organically between
humans, it’s notoriously difficult for human-AI teams, particularly when it
comes to physical robots acquiring knowledge from situated interactions
involving intricate language and physical activities.

To address this challenge, this paper takes a first step towards collaborative
plan acquisition (CPA), where humans and agents aim to communicate, learn, and
infer a complete plan for joint tasks through situated dialogue. To this end,
we extended MindCraft Bara et al. (2021), a benchmark for symmetric
collaborative tasks with disparate knowledge and skills in a 3D virtual world.
Specifically, we formulate a new problem for agents to predict the absent task
knowledge for themselves and for their partners based on a wealth of
perceptual and dialogue history. We start by annotating fine-grained dialogue
moves, which capture the communicative intentions between partners during
collaboration. Our hypothesis is that understanding communicative intentions
plays a crucial role in ToM modeling, which, in turn, facilitate the
acquisition of collaborative plans. We developed a sequence model that takes
the interaction history as input and predicts the dialogue moves, the
partner’s mental states, and the complete plan. Our empirical results suggest
that predicting the partner’s missing knowledge is a more viable approach than
predicting one’s own. We show that explicit modeling of the partner’s dialogue
moves and mental states produces improved and more stable results than
without.

The contributions of this work lie in that it bridges collaborative planning
with situated dialogue to address how partners in a physical world can
collaborate to arrive at a joint plan. In particular, it formulates a novel
task on missing knowledge prediction and demonstrates that it’s feasible for
agents to predict their partner’s missing knowledge with respect to their own
partial plan. Our results have shown that, in human-AI collaboration, a more
viable collaboration strategy is to infer and tell the partner what knowledge
they might be missing and prompt the partner for their own missing knowledge.
This strategy, if adopted by both agents, can potentially improve common
ground in collaborative tasks. Our findings will provide insight for
developing embodied AI agents that can collaborate and communicate with humans
in the future.

##  2 Related Work

Our work bridges several research areas, particularly in the intersection of
human-robot collaboration, planning, and theory of mind modeling.

###  2.1 Mixed-Initiative Planning

The essence of a collaborative task is that two participants, human or
autonomous agents, pool their knowledge and skills to achieve a common goal. A
mixed-initiative planning system involves a human planner and an automated
planner with a goal to reduce the load and produce better plans Queiroz Lino
et al. (2005). We notice an implication that the agent’s functional role is a
supportive one. Work in this paradigm, called intelligent decision support,
involves agents that range from low-level processing and/or visualization
Queiroz Lino et al. (2005) to offering higher-level suggestions Manikonda et
al. (2014). Examples on this spectrum provide agents checking constraint
satisfaction, eliciting user feedback on proposed plans, and ins some cases
the agents are the primary decision makers Zhang et al. (2012); Gombolay et
al. (2015); Sengupta et al. (2017). Our desire is to have human-robot
collaboration starting from an equal footing and a key goal of this is to
resolve disparities in starting knowledge and abilities. We believe this can
be learned from observing human-human interaction and will lead to better
quality collaboration. Prior work indicates that mutual understanding and plan
quality can be improved between intelligent agents through interaction Kim and
Shah (2016), though most of the results are qualitative Di Eugenio et al.
(2000), makes abstract implications Kim and Shah (2016), are not tractable
Grosz and Kraus (1996). Hybrid probabilistic generative and logic-based models
that overcome incomplete information and inconsistent observations have been
proposed by Kim et al. Kim et al. (2015). These were successfully applied to
observe natural human team planning conversations and infer the agreed-upon
plan. Following these footprints, we introduce a collaborative plan
acquisition task to explicitly tackle the initial disparities in knowledge and
abilities of a human planner and an automated planner.

###  2.2 Goal and Plan Recognition

Goal recognition (GR) Heinze (2004); Han and Pereira (2013) refers to the
problem of inferring the goal of an agent based on the observed actions and/or
their effects. On top of that, plan recognition (PR) Kautz et al. (1986);
Carberry (2001) further challenges AI agents to construct a complete plan by
defining a structure with the set of observed and predicted actions that will
lead to the goal Sukthankar et al. (2014); Van-Horenbeke and Peer (2021). We
introduce a collaborative plan acquisition (CPA) task as a step forward along
this line. In the CPA setting, humans and agents start both with incomplete
task knowledge, communicate with each other to acquire a complete plan, and
actively act in a shared environment for joint tasks. We further discuss some
key benefits of our setting compared to existing work. In terms of experiment
setup, the majority of the current approaches employ plan libraries with
predefined sets of possible plans Avrahami-Zilberbrand and Kaminka (2005);
Mirsky et al. (2016), or domain theories to enable plan recognition as
planning (PRP) Ramírez and Geffner (2009); Sohrabi et al. (2016), which suffer
from scalability issue in complex domains Pereira et al. (2017) for high-
dimensional data Amado et al. (2018). Motivated by existing research Rabkina
et al. (2020, 2021), we adapt Minecraft as our planning domain, as it allows
us to define agents with hierarchical plan structures and visual perception in
a 3D block world that requires plan recognition from latent space. In terms of
task setup, the setting of the CPA task shares the merit of active goal
recognition Shvo and McIlraith (2020), where agents are not passive observers
but are enabled to sense, reason, and act in the world. We further enable
agents to communicate with their partners with situated dialogue, which is
more realistic in real-world human-robot interaction. Although there exists
research to integrate non-verbal communication to deal with incomplete plans
in sequential plan recognition Mirsky et al. (2016, 2018) and research to
integrate natural language processing through parsing Geib and Steedman
(2007), little work was done to explore language communication and dialogue
processing. The CPA task introduces a more general and symmetric setting,
where agents not only query their partners for missing knowledge but also
actively share knowledge that their partners may be ignorant of.

###  2.3 Theory of Mind Modeling

As introduced by Premack and Woodruff Premack and Woodruff (1978), one has a
Theory of Mind (ToM) if they impute mental states to themselves and others.
While interacting with others, Humans use their ToM to predict partners’
future actions Dennett (1988), to plan to change others’ beliefs and next
actions Ho et al. (2022) and to facilitate their own decision-making Rusch et
al. (2020). In recent years, the AI community has made growing efforts to
model a machine ToM to strengthen agents in human-robot interaction Krämer et
al. (2012) and multiagent coordination Albrecht and Stone (2018). We compare
our work with representative work along this line in two dimensions. In terms
of the role of the agent, prior research is largely limited to passive
observer roles Grant et al. (2017); Nematzadeh et al. (2018); Le et al.
(2019); Rabinowitz et al. (2018) or as a speaker in a Speaker-Listener
relationship Zhu et al. (2021). Following a symmetric and collaborative setup
Bara et al. (2021); Sclar et al. (2022), we study ToM modeling in agents that
actively interact with the environment and engage in free-form situated
communication with a human partner. In terms of task formulation, machine ToM
has been typically formulated as inferring other agents’ beliefs Grant et al.
(2017); Nematzadeh et al. (2018); Le et al. (2019); Bara et al. (2021),
predicting future actions Rabinowitz et al. (2018), generating pragmatic
instructions Zhu et al. (2021), and gathering information Sclar et al. (2022).
None of these formulations were able to explicitly assess how well can AI
agents use their machine ToM to complete partial plans through situation
communication with their collaborators, as humans usually do in real-world
interactions. To this end, we extended the problem formulation in Bara et al.
(2021) to a collaborative plan acquisition task, where humans and agents try
to learn and communicate with each other to acquire a complete plan for joint
tasks.

Figure 1: An example dialogue history between two partners to complete a joint
goal in MindCraft. Each player is given a partial plan. They communicate with
each other to form a complete plan for the goal. Each utterance is annotated
with a dialogue move that describes the communicative intention.

##  3 Background: ToM for Collaborative Tasks

Figure 2: An example of the plan graphs. From left to right, we illustrate a
complete joint plan, a partial plan for Agent A, and another partial plan for
Agent B. The plan graphs all contain the same set of nodes, with a joint goal
$a$ on top and other nodes representing fact landmarks. In a collaborative
plan acquisition problem, an agent is tasked to infer its own missing
knowledge and its partner’s missing knowledge.

Our work is built upon MindCraft, a benchmark and platform developed by Bara
et al. (2021) for studying ToM modeling in collaborative tasks. We first give
a brief introduction to this benchmark and then illustrate how our work
differs from MindCraft.111The original dataset consists of 100 dialogue
sessions. We used the platform and collected 60 additional sessions to
increase the data size for our investigation under the IRB (HUM00166817)
approved by the University of Michigan.

The MindCraft platform supports agents to complete collaborative tasks through
situated dialogue in a 3D block world, with rich mental state annotations. As
shown in Figure 1, two agents are co-situated in a shared environment and
their joint goal is to create a specific material. There are two macro-
actions: (1) creating a block and (2) combining two existing blocks to create
a new block. These macro-actions are made up of atomic actions that agents may
perform in-game, e.g., navigation, jumping, and moving blocks around. During
gameplay, the two agents are each given a partial plan in the form of a
directed AND-graph. These partial plans are incomplete in the sense that the
agents cannot complete the task by themselves individually by following the
plan. The two players will need to communicate with each other through the in-
game chat so that they can coordinate with each other to complete the joint
task. An example dialogue session between the players to communicate the plan
is shown in Figure 1.

As an initial attempt, Bara et al. Bara et al. (2021) formulated ToM modeling
as three tasks that predict a partner’s mental state:

  * •

Task Intention: predict the sub-goal that the partner is currently working on;

  * •

Task Status: predict whether the partner believes a certain sub-goal is
completed and by whom;

  * •

Task Knowledge: predict whether the partner knows how to achieve a sub-goal,
i.e., all the incoming edges of a node;

A baseline was almost implemented that takes perceptual observation and
interaction history for these prediction tasks and reported results in Bara et
al. (2021). For the remainder of this paper, we use ToM tasks to refer to
these three tasks introduced in the original paper.

It’s important to note that, although we use the MindCraft benchmark, our work
here has several significant differences. First and foremost, while MindCraft
studies ToM, it primarily focused on inferring other agents’ mental states,
and has not touched upon collaborative plan acquisition. How humans and agents
communicate, learn, and infer a complete plan for joint tasks through situated
dialogue is the new topic we attempt to address in this paper. Second,
MindCraft mostly focuses on ToM modeling and only provides ground-truth labels
for the three tasks described above. As communicative intentions play an
important role in coordinating activities between partners, we added
additional annotations for dialogue moves (as shown in Figure 1). We
investigate if the incorporation of dialogue moves would benefit mental state
prediction and plan acquisition.

##  4 Collaborative Plan Acquisition

In a human-AI team like that in MindCraft, humans and AI agents may have
insufficient domain knowledge to derive a complete plan, thus suffering from
an incomplete action space to execute a complete plan. It’s therefore
important for an agent to predict what knowledge is missing for themselves and
for their partners, and proactively seek/share that information so the team
can reach a common and complete plan for the joint goal. We start by
formalizing the plan representation, followed by a description of the
collaborative plan acquisition problem.

###  4.1 Task Formulation

######  Definition 1 (Joint and Partial Plan).

We represent a joint plan $\mathcal{P}=(V,E)$ as a directed AND-graph, where
the nodes $V$ denote sub-goals and the edges $E$ denote temporal constraints
between the subgoals. As a directed AND-graph, all of the children sub-goals
of an AND-node must be satisfied in order to perform the parent. A partial
plan $\tilde{\mathcal{P}}=(V,\tilde{E})$ is a subgraph of $\mathcal{P}$ with a
shared set of nodes $V$ but only share a subset of edges $\tilde{E}\subseteq
E$.

An example of a complete plan graph in MindCraft is shown in Figure 2. Each
plan contains a joint goal, and the rest of the nodes denotes fact landmarks
Hoffmann et al. (2004), i.e., sub-goals that must be achieved at some point
along all valid execution.

We consider a pair of collaborative agents with a joint plan $\mathcal{P}$. To
account for the limited domain knowledge, an agent $i$ has an initial partial
plan $\mathcal{P}_{i}=(V,E_{i}),E_{i}\subseteq E$, which is a subgraph of
$\mathcal{P}$. As shown in Figure 2, the complete plan and partial plans share
the same set of nodes (i.e., the same $V$). The agent only has access to its
own knowledge, which might be shared (denoted as blue arrows) or not (denoted
as green arrows). Its missing knowledge (denoted as grey arrows) is a set of
edges $\overline{E}_{i}=E\backslash E_{i}$. We assume, in this work, the
collaborative planning problem is solvable for the collaborative agents, i.e.,
$\bigcup E_{i}=E$. In order for the agents to solve the problem, agents can
communicate with their partners by sending a natural language message, which
in turn helps them to acquire a complete plan graph. We define this
collaborative plan acquisition problem formally as follows.

######  Definition 2 (Collaborative Plan Acquisition Problem).

In a collaborative plan acquisition problem with a joint plan $\mathcal{P}$,
an agent $i$ and its collaborative partner $j$ start with partial plans
$\mathcal{P}_{i}=(V,E_{i})$ and $\mathcal{P}_{j}=(V,E_{j})$. At each timestamp
$t$, the agent $i$ has access to a sequence of up-to-date visual observations
$O_{i}^{(t)}$ and dialogue history $D^{(t)}$. The problem is for agent $i$’s
to acquire its own missing knowledge $\overline{E}_{i}=E\backslash E_{i}$ and
the partner $j$’s missing knowledge $\overline{E}_{j}=E\backslash E_{j}$.

To solve a collaborative plan acquisition problem, agent $i$ needs to address
two tasks.

##### Task 1: Inferring Own Missing Knowledge.

For Task 1, a solution is a set of missing knowledge
$\overline{E}_{i}=E\backslash E_{i}$. Agent $i$ needs to infer its own missing
knowledge by identifying the missing edges in its partial plan from the
complete joint plan, i.e., $P(e\in E\backslash E_{i}\ |\
O_{i}^{(t)},D_{i}^{(t)}),\forall e\in V^{2}\backslash E_{i}$ at time $t$. Note
that $V^{2}$ refers to a complete graph, instead of a complete joint plan. For
example, as shown in Figure 2, among all the missing edges in Agent A’s
partial plan, we hope Agent A would correctly predict that the edge
$d\rightarrow c$ and the edge $e\rightarrow c$ are missing in their own plan.
Recovering those edges leads to a complete joint plan.

##### Task 2: Inferring Partner’s Missing Knowledge.

For Task 2, a solution is a set of missing knowledge
$\overline{E}_{j}=E\backslash E_{j}$. Agent $i$ predicts what edges in $i$’s
partial plan which might be missing in its partner $j$’s partial plan, i.e.,
$P(e\in E\backslash E_{j}\ |\ O_{i}^{(t)},D_{i}^{(t)}),\forall e\in E_{i}$ at
time $t$. In the example in Figure 2, Agent A should select the edges
$e\rightarrow d$ and $f\rightarrow d$ from their own partial plan as being
absent from their partner’s plan. If the agent can correctly predict which
edges are missing for their partner, the agent can proactively communicate to
their partner and therefore help their partner acquire a complete task plan.
If agents can predict what each other is missing and proactively share that
knowledge, then both agents will be able to reach a common understanding of
the complete joint plan.

We note that the Task Knowledge in ToM tasks is different from the Task 2 we
propose. In Task Knowledge, the model is probed whether one piece of knowledge
that might be unknown itself is known by the partner. In Task 2, the model
needs to predict, for each piece of the agent’s own knowledge, whether the
partner shares it or not.

Figure 3: The theory of mind (ToM) model consists of a base sequence model
taking in as input representations for dialogue ($D$) when available), visual
observation of the environment ($O$), and the partial plan available to the
agent. The model can be configured to take optional inputs as latent
representations from the frozen mental state prediction models and the
dialogue move representation for dialogue exchanges.

###  4.2 Dialogue Moves in Coordination of Plans

While the ToM tasks have captured the partner’s task intention, another
important dimension of intention, the communicative intention, was neglected
in the original MindCraft paper. This can be captured by dialogue moves, which
are sub-categories of dialogue acts that guide the conversation and update the
information shared between the speakers Traum and Larsson (2003); Marge et al.
(2020). To this end, we introduce a dialogue move prediction task to better
understand the dialogue exchanges between the agent and its partner in a
collaborative planning problem. We build our move schema from the set of
dialogue acts described in Stolcke et al. (2000), out of which we keep a
relevant subset. We expand the Directive, Statement, and Inquiry categories
for domain-specific uses in the MindCraft collaborative task. For these three
categories, we introduce parameters that complete the semantics of dialogue
moves. These parameters serve the purpose of grounding the dialogue to the
partial plan graph given to the player. We show all dialogue moves with
parameters in Table 6 in Section A.1 in the supplemental material.

###  4.3 Computational Model

##### End-to-End Baseline.

We start by introducing a straightforward end-to-end approach to address both
Task 1 and Task 2, similar to the model in Bara et al. (2021) for consistency.
More specifically, the baseline model processes the dialogue input with a
frozen language model Devlin et al. (2019), the video frames with a
convolutional neural network, and the partial plan with a gated recurrent unit
Chung et al. (2014). The sequences of visual, dialogue, and plan
representations are fed into an LSTM Hochreiter and Schmidhuber (1997). The
predictions of the missing knowledge of the agent and its partner are decoded
with feed-forward networks.

##### Augmented Models.

One important research question we seek to address is whether explicit
treatment of the partner’s mental states and dialogue moves would benefit
collaborative plan acquisition. As shown in Figure 3, we develop a multi-stage
augmented model that first learns to predict the dialogue moves, then attempts
to model a theory of mind, and finally learns to predict the missing knowledge
of itself and of its partner respectively. The model processes each input
modality and time series with the same architectural design as the baseline.
At Stage 1, the models predict the dialogue moves of the partner. At Stage 2,
we freeze the model pre-trained in Stage 1 and concatenate their output latent
representations to augment the input sequences of visual, dialogue, and plan
representations. The models predict the mental states of the partner, with
each LSTM sequence model dedicated to task intention, task status, and task
knowledge. At Stage 3, we freeze the models pre-trained in Stages 1 and 2 and
task the model to predict the missing knowledge with similar LSTM and feed-
forward networks. 222Our code is available at https://github.com/sled-
group/collab-plan-acquisition.

##  5 Empirical Studies

In this section, we first examine the role of dialogue moves in three ToM
tasks and further discuss how dialogue moves and ToM modeling influence the
quality of collaborative plan acquisition.

###  5.1 Role of Dialogue Moves in ToM Tasks

The performance of the mental state prediction models, as presented in
MindCraft, shows low performance in the multimodal setting. We begin by
confirming the effectiveness of dialogue moves, by evaluating if they help to
improve these ToM tasks proposed in Bara et al. (2021). The same setting as
described by the Stage 1 and 2. We show results in Table 1, which compares the
performance of the baseline model with the model augmented with dialogue
moves. We observe a significant increase in performance when using dialogue
moves. Furthermore, for the task of predicting the partner’s task knowledge,
we observe that the augmented model approaches the average human
performance.333The average human performance measure was provided in the
original MindCraft Bara et al. (2021). The best-performing models for every
ToM task are used to produce the latent representations for our subsequent
tasks.

Task Status
---
Modalities | w/o Dlg Moves | w/ Dlg Moves | Human
None | N/A | 56.0$\pm 0.8$ | 67.0
D | 45.8$\pm 3.0$ | 54.6$\pm 1.1$ | 67.0
D+O | 32.7$\pm 1.2$ | 59.3$\pm 1.0$ | 67.0
O | 53.7$\pm 1.1$ | 59.3$\pm 1.7$ | 67.0
Task Knowledge
Modalities | w/o Dlg Moves | w/ Dlg Moves | Human
None | N/A | 54.7$\pm 2.5$ | 58.0
D | 45.3$\pm 1.3$ | 56.2$\pm 1.9$ | 58.0
D+O | 48.3$\pm 1.1$ | 57.6$\pm 1.0$ | 58.0
O | 49.4$\pm 2.5$ | 56.4$\pm 2.5$ | 58.0
Task Intention
Modalities | w/o Dlg Moves | w/ Dlg Moves | Human
None | N/A | 14.9$\pm 1.5$ | 46.0
D | 3.0$\pm 0.6$ | 12.1$\pm 1.0$ | 46.0
D+O | 6.2$\pm 0.6$ | 13.5$\pm 0.6$ | 46.0
O | 6.6$\pm 1.1$ | 13.8$\pm 1.7$ | 46.0

Table 1: Performance on the three ToM tasks with or without using dialogue
moves. None means only dialogue moves are used for prediction. D stands for
text in dialogue; O stands for visual observation of the activities in the
environment; D+O both text and visual observation. Highlighted values are
statistically significant with $P<0.01$ compared to the best model without
dialogue moves for the given task.

###  5.2 Results for Collaborative Plan Acquisition

We now present the empirical results for collaborative plan acquisition, i.e.,
inferring one’s own missing knowledge and inferring the partner’s missing
knowledge at the end of each session. We use the following metrics to evaluate
the performance on these tasks:

  * •

Per Edge F1 Score, computed by aggregating all edges across tasks. It is meant
to evaluate the model’s ability to predict whether an edge is missing in a
partial plan.

  * •

Per Task F1 Score, computed as the average of F1 scores within a dialogue
session. It is meant to evaluate the model’s average performance across
sessions.

##### Task 1: Inferring Own Missing Knowledge.

The performance is shown in Table 2. Overall, we found the models underperform
across all configurations, meaning that inferring one’s own missing knowledge
turns out to be a difficult task. We believe this is due to the sparsity of
the task graph. Since the space of possible edges to be predicted is large (as
the agent needs to consider every possible link between two nodes), the link
prediction becomes notoriously challenging. Better solutions will be needed
for this task in the future.

Task | Task |  Task. |  Dlg. | Per Edge | Per Task
---|---|---|---|---|---
Status |  Know. | Int. | Move | F1 Score | F1 Score
|  |  |  | 17.0 $\pm$ 0.2 | 19.8 $\pm$ 1.0
| X |  |  | 19.0 $\pm$ 2.5 | 21.1 $\pm$ 1.6
|  | X |  |  21.0  $\pm$ 0.7 | 22.2 $\pm$ 2.2
| X | X |  |  19.6 $\pm$ 1.4 | 21.4 $\pm$ 1.7
| X |  | X |  20.1 $\pm$ 1.4 | 22.1 $\pm$ 1.2
|  | X | X |  19.8 $\pm$ 1.7 | 21.7 $\pm$ 1.8
X | X | X | X | 17.4 $\pm$ 0.1 | 20.0 $\pm$ 1.9

Table 2: Performance on inferring agent’s own knowledge. Highlighted values
are statistically significant with $P<0.01$ compared to the base model without
augmentation.

Task | Task |  Task. |  Dlg. | Per Edge | Per Task
---|---|---|---|---|---
Status |  Know. | Int. | Move | F1 Score | F1 Score
|  |  |  | 71.3 $\pm$ 1.1 | 68.8 $\pm$ 3.1
| X |  |  |  74.4 $\pm$ 0.3 | 73.6 $\pm$ 1.4
|  | X |  |  74.5 $\pm$ 1.4 | 73.8 $\pm$ 3.0
| X | X |  |  74.4 $\pm$ 1.4 | 73.5 $\pm$ 2.3
| X |  | X |  74.3 $\pm$ 0.7 |  73.5 $\pm$ 1.3
|  | X | X |  75.0 $\pm$ 1.0 |  74.7 $\pm$ 2.2
X | X | X | X |  73.5 $\pm$ 0.5 | 72.1 $\pm$ 1.8

Table 3: Performance on inferring the player’s partner’s plan. Highlighted
values are statistically significant with $P<0.01$ compared to the base model
without augmentation.

##### Task 2: Inferring Partner’s Missing Knowledge.

Table 3 shows the performance which is across the board more than 70% F1. This
means that this task is more approachable compared to inferring one’s own
missing knowledge. Table 3 also compares various combinations of augmented
models, with different augmentations available from Stage 1 (e.g., latent
representations of dialogue moves and other mental states). While all
combinations lead to increased performance, we found in general that
incorporating dialogue moves and task intention has the highest performance.
This finding confirms the importance of intention prediction in plan
coordination, from both the communication level and task level.

###  5.3 Cross-time Analysis

(a) Per edge F1 score

(b) Per task F1 score

Figure 4: Model performance as the interaction progresses in absolute time.
The abbreviations are S - Task Status, K - Task Knowledge, I - Task Intent,
and DM - Dialogue Moves.

(a) Cumulative number of times the model changes its mind.

(b) Cumulative absolute change in confidence.

Figure 5: The average change in model prediction over each edge as the
interaction progresses in absolute time. The cause for the curve’s dips is
that not all interactions are of equal length. The further in time, the fewer
interactions there are that have data at that time step. The abbreviations are
S - Task Status, K - Task Knowledge, I - Task Intent, and DM - Dialogue Moves.

As Task 2 is more approachable with more reasonable performance, we look
further into the performance of the models as the interaction progresses. In
this section, we look at not only the latent representation at the end of an
interaction but also throughout the interaction, to predict the partner’s
missing knowledge. We find that as the interaction progresses, in absolute
time, there is an upward trend concerning both per edge F1 score (Figure
4(a)), as well as concerning per task F1 score (Figure 4(b)). This is not
surprising. As models have access to richer interaction history, which enables
them to make more reliable predictions of the partner’s missing knowledge.
Furthermore, we observe that the augmented models improve the performance
across time, confirming the results we obtained in the previous section.

We are also interested in the stability of system prediction over time, which
is an important feature if these agents are ever employed in real interaction
with humans. We introduce two metrics for this purpose:

  * •

Number of Mind Changes, which is the number of times that the model changes
its prediction of whether an edge is missing from its partner’s knowledge.

  * •

Accumulated Absolute Confidence Changes, which adds up the absolute value of
the changes in the prediction probability between timestamps.

Figure 5(a) shows the average number of mind changes and Figure 5(b) shows the
accumulated absolute confidence changes as the interaction progresses. We
observe that the augmented models show significantly fewer changes in
prediction and a lower change in prediction confidence, as compared with the
base model. These results have shown that the base model is more inclined to
“change its mind” in prediction as interaction proceeds, the augmented models
that take into account of partner’s mental states and dialogue moves are more
stable in their prediction throughout the interaction.

##  6 Discussion and Conclusion

In this work, we address the challenge of collaborative plan acquisition in
human-agent collaboration. We extend the MindCraft benchmark and formulate a
problem for agents to predict missing task knowledge based on perceptual and
dialogue history, focusing on understanding communicative intentions in Theory
of Mind (ToM) modeling. Our empirical results highlight the importance of
predicting the partner’s missing knowledge and explicitly modeling their
dialogue moves and mental states. A promising strategy for effective
collaboration involves inferring and communicating missing knowledge to the
partner while prompting them for their own missing knowledge. This
collaborative approach holds the potential to improve decision-making when
both agents actively engage in its implementation. The findings have
implications for the development of embodied AI agents capable of seamless
collaboration and communication with humans. Specifically, by predicting their
partner’s missing knowledge and actively sharing that information, these
agents can facilitate a shared understanding and successful execution of joint
tasks. The future efforts following this research could explore and refine
this collaborative strategy.

This work presents our initial results. It also has several limitations. The
current setup assumes shared goals and a single optimal complete plan without
alternatives, neglecting the complexity that arises from the absence of shared
goals and the existence of alternative plans. Our motivation for controlling
the form of partial plans and the predetermined complete plan is to enable a
systematic focus on modeling and evaluating plan coordination behaviors.
Although our current work is built on the MindCraft dataset where partial
plans are represented by AND-graphs, the problem formulation can be
potentially generalized to multiple AND-OR-graphs. Future research could
explore approaches that incorporate AND-OR-graphs to account for alternative
paths to achieving joint goals.

Additionally, our present study focuses on a dyadic scenario, employing human-
human collaboration data to study collaborative plan acquisition. Since AI
agents typically have limited visual perception and reasoning abilities
compared to their human counterparts, the communication discourse is expected
to exhibit increased instances of confirmations, repetitions, and corrections.
How to effectively extend the models trained on human-human data to human-
agent collaboration remains an important question. With the emergence of large
foundation models Bommasani et al. (2021), our future work will incorporate
these models into our framework to facilitate situated dialogue for
collaborative plan acquisition. We will further conduct experiments and
evaluate the efficacy of these models in more complex human-agent
collaborations.

##  7 Acknowledgments

This work was supported in part by NSF IIS-1949634 and NSF SES-2128623. The
authors would like to thank the anonymous reviewers for their valuable
feedback.

## References

  * Albrecht and Stone [2018] Stefano V Albrecht and Peter Stone.  Autonomous agents modelling other agents: A comprehensive survey and open problems.  Artificial Intelligence, 258:66–95, 2018.
  * Amado et al. [2018] Leonardo Rosa Amado, João Paulo Aires, RAMON FRAGA PEREIRA, Maurício Cecílio Magnaguagno, Roger Leitzke Granada, and Felipe Rech Meneguzzi.  Lstm-based goal recognition in latent space.  In ICML/IJCAI/AAMAS 2018 Workshop on Planning and Learning (PAL-18), 2018, Suécia., 2018.
  * Avrahami-Zilberbrand and Kaminka [2005] Dorit Avrahami-Zilberbrand and Gal A Kaminka.  Fast and complete symbolic plan recognition.  In IJCAI, pages 653–658, 2005.
  * Bara et al. [2021] Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai.  MindCraft: Theory of mind modeling for situated dialogue in collaborative tasks.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1112–1125, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, et al.  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258, 2021.
  * Carberry [2001] Sandra Carberry.  Techniques for plan recognition.  User modeling and user-adapted interaction, 11(1):31–48, 2001.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555, 2014.
  * Dennett [1988] Daniel C Dennett.  Précis of the intentional stance.  Behavioral and brain sciences, 11(3):495–505, 1988.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * Di Eugenio et al. [2000] Barbara Di Eugenio, Pamela W Jordan, Richmond H Thomason, and JOHANNA D MOORE.  The agreement process: An empirical investigation of human–human computer-mediated collaborative dialogs.  International Journal of Human-Computer Studies, 53(6):1017–1076, 2000.
  * Geib and Steedman [2007] Christopher W Geib and Mark Steedman.  On natural language processing and plan recognition.  In IJCAI, volume 2007, pages 1612–1617, 2007.
  * Gombolay et al. [2015] Matthew C Gombolay, Reymundo A Gutierrez, Shanelle G Clarke, Giancarlo F Sturla, and Julie A Shah.  Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams.  Autonomous Robots, 39(3):293–312, 2015.
  * Grant et al. [2017] Erin Grant, Aida Nematzadeh, and Thomas L Griffiths.  How can memory-augmented neural networks pass a false-belief task?  In CogSci, 2017.
  * Grosz and Kraus [1996] Barbara J Grosz and Sarit Kraus.  Collaborative plans for complex group action.  Artificial Intelligence, 86(2):269–357, 1996.
  * Han and Pereira [2013] The Anh Han and Luís Pereira.  State-of-the-art of intention recognition and its use in decision making: A research summary.  AI Communications, 26(2):237–246, 2013.
  * Heinze [2004] Clint Heinze.  Modelling intention recognition for intelligent agent systems.  Technical report, DEFENCE SCIENCE AND TECHNOLOGY ORGANISATION SALISBURY (AUSTRALIA) SYSTEMS …, 2004.
  * Ho et al. [2022] Mark K Ho, Rebecca Saxe, and Fiery Cushman.  Planning with theory of mind.  Trends in Cognitive Sciences, 2022.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * Hoffmann et al. [2004] Jörg Hoffmann, Julie Porteous, and Laura Sebastia.  Ordered landmarks in planning.  Journal of Artificial Intelligence Research, 22:215–278, 2004.
  * Kautz et al. [1986] Henry A Kautz, James F Allen, et al.  Generalized plan recognition.  In AAAI, volume 86, page 5. Philadelphia, PA, 1986.
  * Kim and Shah [2016] Joseph Kim and Julie A Shah.  Improving team’s consistency of understanding in meetings.  IEEE Transactions on Human-Machine Systems, 46(5):625–637, 2016\.
  * Kim et al. [2015] Been Kim, Caleb M Chacha, and Julie A Shah.  Inferring team task plans from human meetings: A generative modeling approach with logic-based prior.  Journal of Artificial Intelligence Research, 52:361–398, 2015.
  * Krämer et al. [2012] Nicole C Krämer, Astrid von der Pütten, and Sabrina Eimler.  Human-agent and human-robot interaction theory: Similarities to and differences from human-human interaction.  In Human-computer interaction: The agency perspective, pages 215–240. Springer, 2012.
  * Le et al. [2019] Matthew Le, Y-Lan Boureau, and Maximilian Nickel.  Revisiting the evaluation of theory of mind through question answering.  In EMNLP-IJCNLP 2019, pages 5872–5877, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * Manikonda et al. [2014] Lydia Manikonda, Tathagata Chakraborti, Sushovan De, Kartik Talamadupula, and Subbarao Kambhampati.  Ai-mix: using automated planning to steer human workers towards better crowdsourced plans.  In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.
  * Marge et al. [2020] Matthew Marge, Felix Gervits, Gordon Briggs, Matthias Scheutz, and Antonio Roque.  Let’s do that first! a comparative analysis of instruction-giving in human-human and human-robot situated dialogue.  In Proceedings of the 24th Workshop on the Semantics and Pragmatics of Dialogue (SemDial), 2020.
  * Mirsky et al. [2016] Reuth Mirsky, Roni Stern, Ya’akov Gal, and Meir Kalech.  Sequential plan recognition.  In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 401–407, 2016.
  * Mirsky et al. [2018] Reuth Mirsky, Roni Stern, Kobi Gal, and Meir Kalech.  Sequential plan recognition: An iterative approach to disambiguating between hypotheses.  Artificial Intelligence, 260:51–73, 2018.
  * Nematzadeh et al. [2018] Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths.  Evaluating theory of mind in question answering.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2392–2400, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  * Pereira et al. [2017] Ramon Pereira, Nir Oren, and Felipe Meneguzzi.  Landmark-based heuristics for goal recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  * Premack and Woodruff [1978] David Premack and Guy Woodruff.  Does the chimpanzee have a theory of mind?  Behavioral and Brain Sciences, 1(4):515–526, 1978.
  * Queiroz Lino et al. [2005] Natasha Queiroz Lino, Austin Tate, and Yun-Heh Chen-Burger.  Semantic support for visualisation in collaborative ai planning.  In Susanne Biundo, Karen Myers, and Kanna Rajan, editors, Proceedings of the Workshop on The Role of Ontologies in Planning and Scheduling. AAAI Press, June 2005.  The Fifteenth International Conference on Automated Planning and Scheduling, ICAPS.
  * Rabinowitz et al. [2018] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick.  Machine theory of mind.  In International conference on machine learning, pages 4218–4227. PMLR, 2018.
  * Rabkina et al. [2020] Irina Rabkina, Pavan Kathnaraju, Mark Roberts, Jason Wilson, Kenneth Forbus, and Laura Hiatt.  Recognizing the goals of uninspectable agents.  In Proceedings of the Conference on Advances in Cognitive Systems (Palo Alto, CA:), pages 1–8, 2020.
  * Rabkina et al. [2021] Irina Rabkina, Pavan Kantharaju, Jason R Wilson, Mark Roberts, and Laura M Hiatt.  Evaluation of goal recognition systems on unreliable data and uninspectable agents.  Frontiers in Artificial Intelligence, 4, 2021.
  * Ramírez and Geffner [2009] Miquel Ramírez and Hector Geffner.  Plan recognition as planning.  In Twenty-First international joint conference on artificial intelligence, 2009.
  * Rusch et al. [2020] Tessa Rusch, Saurabh Steixner-Kumar, Prashant Doshi, Michael Spezio, and Jan Gläscher.  Theory of mind and decision science: towards a typology of tasks and computational models.  Neuropsychologia, 146:107488, 2020.
  * Sclar et al. [2022] Melanie Sclar, Graham Neubig, and Yonatan Bisk.  Symmetric machine theory of mind.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 19450–19466, 17–23 Jul 2022.
  * Sengupta et al. [2017] Sailik Sengupta, Tathagata Chakraborti, Sarath Sreedharan, Satya Gautam Vadlamudi, and Subbarao Kambhampati.  Radar-a proactive decision support system for human-in-the-loop planning.  In 2017 AAAI Fall Symposium Series, 2017.
  * Shvo and McIlraith [2020] Maayan Shvo and Sheila A McIlraith.  Active goal recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9957–9966, 2020.
  * Sohrabi et al. [2016] Shirin Sohrabi, Anton V Riabov, and Octavian Udrea.  Plan recognition as planning revisited.  In IJCAI, pages 3258–3264. New York, NY, 2016.
  * Stolcke et al. [2000] Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer.  Dialogue act modeling for automatic tagging and recognition of conversational speech.  Computational linguistics, 26(3):339–373, 2000.
  * Sukthankar et al. [2014] Gita Sukthankar, Christopher Geib, Hung Bui, David Pynadath, and Robert P Goldman.  Plan, activity, and intent recognition: Theory and practice.  Newnes, 2014.
  * Traum and Larsson [2003] David R Traum and Staffan Larsson.  The information state approach to dialogue management.  In Current and new directions in discourse and dialogue, pages 325–353. Springer, 2003.
  * Van-Horenbeke and Peer [2021] Franz A Van-Horenbeke and Angelika Peer.  Activity, plan, and goal recognition: A review.  Frontiers in Robotics and AI, 8:643010, 2021.
  * Zhang et al. [2012] Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz.  Human computation tasks with global constraints.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 217–226, 2012.
  * Zhu et al. [2021] Hao Zhu, Graham Neubig, and Yonatan Bisk.  Few-shot language coordination by modeling theory of mind.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12901–12911, 18–24 Jul 2021.

##  Appendix A Appendix

###  A.1 Dialogue Move Taxonomy and Annotation

The Dialogue Move consists of a dialogue category with up to three value slots
which are representations of materials, mines, or tools used in the
collaborative task. Table 6 shows the full set of Dialogue moves with the type
and number of parameters where applicable. The distribution of the dialogue
move categories is presented in Figure 6. The dialogue moves were labeled by a
group of 7 human annotators. The dataset was divided into eight parts with one
part being given to all annotators. The Cohen’s kappa score, calculated on the
common part of the dataset, between the annotators, if $k=0.807$ which is
considered a strong agreement.

Figure 6: Distribution of Dialogue Moves in our Dataset.

###  A.2 Model Description

####  A.2.1 Input Modalities Processing

Input processing of dialogue utterances, video frames, and plan representation
is processed identically to MindCraft Bara et al. [2021]: dialogue utterances
through BERT, video frames through a CNN, and partial plans through a GRU.

####  A.2.2 Dialogue Moves Encoding

The dialogue move representations are groups of four one-hot embeddings
representing the dialogue move type, see Table 6, followed three slot
embeddings which are either a material, a tool, a mine, or to be ignored.

####  A.2.3 Augmented Model

The augmented models utilize an LSTM as a sequence model to process the time
series. On top of the three input modalities, there are four more optional
inputs: a latent representation at every frame of when the partner’s task
status, task knowledge, task intention, or the dialogue move if one is
available. The latent representations for the ToM tasks are retrieved from the
output of the sequence model before the feed-forward networks.

####  A.2.4 Computation Cost

Our experiments were performed on an NVIDIA RTX A6000. It takes around 60 min
per experiment to train the model. At test time, it takes 82 ms on average per
inference.

###  A.3 Detailed Plan Prediction Results

Table 4 shows the change in performance for the task of predicting the
player’s own missing knowledge with all configurations of the augmented
models.

Task | Tast |  Task. |  Dlg. | Per Edge | Avg. Per Game
---|---|---|---|---|---
Status |  Know. | Int. | Move | F1 Score | F1 Score
|  |  |  | 17.0 $\pm$ 0.2 | 19.8 $\pm$ 1.0
X |  |  |  | 20.4 $\pm$ 1.1 | 23.3 $\pm$ 1.5
| X |  |  | 19.0 $\pm$ 2.5 | 21.1 $\pm$ 1.6
X | X |  |  | 20.9 $\pm$ 1.5 | 23.3 $\pm$ 2.2
|  | X |  | 21.0 $\pm$ 0.7 | 22.2 $\pm$ 2.2
X |  | X |  | 19.4 $\pm$ 1.3 | 23.5 $\pm$ 1.3
| X | X |  | 19.6 $\pm$ 1.4 | 21.4 $\pm$ 1.7
X | X | X |  | 18.2 $\pm$ 1.0 | 21.5 $\pm$ 2.4
|  |  | X | 16.7 $\pm$ 0.1 | 19.6 $\pm$ 0.8
X |  |  | X | 20.4 $\pm$ 1.4 | 22.6 $\pm$ 0.8
| X |  | X | 20.1 $\pm$ 1.4 | 22.1 $\pm$ 1.2
X | X |  | X | 20.9 $\pm$ 1.2 | 24.3 $\pm$ 2.1
|  | X | X | 19.8 $\pm$ 1.7 | 21.7 $\pm$ 1.8
X |  | X | X | 19.8 $\pm$ 0.8 | 24.0 $\pm$ 2.9
| X | X | X | 20.3 $\pm$ 1.8 | 22.0 $\pm$ 1.6
X | X | X | X | 17.4 $\pm$ 0.1 | 20.0 $\pm$ 1.9

Table 4: Change in performance in completing agent’s own plan.

Table 5 shows the change in performance for the task of predicting the
partner’s missing knowledge with all augmenting input configurations.

Task | Task |  Task. |  Dlg. | Per Edge | Avg. Per Game
---|---|---|---|---|---
Status |  Know. | Int. | Move | F1 Score | F1 Score
|  |  |  | 71.3 $\pm$ 1.1 | 68.8 $\pm$ 3.1
X |  |  |  | 72.5 $\pm$ 0.5 | 70.7 $\pm$ 2.2
| X |  |  | 74.4 $\pm$ 0.3 | 73.6 $\pm$ 1.4
X | X |  |  | 72.4 $\pm$ 1.7 | 70.4 $\pm$ 3.6
|  | X |  | 74.5 $\pm$ 1.4 | 73.8 $\pm$ 3.0
X |  | X |  | 72.1 $\pm$ 1.5 | 69.6 $\pm$ 3.3
| X | X |  | 74.4 $\pm$ 1.4 | 73.5 $\pm$ 2.3
X | X | X |  | 72.6 $\pm$ 1.7 | 70.2 $\pm$ 4.1
|  |  | X | 71.4 $\pm$ 1.0 | 69.1 $\pm$ 3.1
X |  |  | X | 71.3 $\pm$ 1.6 | 68.8 $\pm$ 4.0
| X |  | X | 74.3 $\pm$ 0.7 | 73.5 $\pm$ 1.3
X | X |  | X | 73.1 $\pm$ 1.5 | 71.2 $\pm$ 3.3
|  | X | X | 75.0 $\pm$ 1.0 | 74.7 $\pm$ 2.2
X |  | X | X | 71.9 $\pm$ 1.5 | 69.3 $\pm$ 3.3
| X | X | X | 73.4 $\pm$ 1.2 | 72.4 $\pm$ 2.0
X | X | X | X | 73.5 $\pm$ 0.5 | 72.1 $\pm$ 1.8

Table 5: Change in performance on completing the partner’s plan.

Name | Parameters | Description or Example
---|---|---
Directive-Make | Block | Directing their partner to make a material
Directive-Other | N/A | Ambiguous directive
Directive-PickUp | Block | Directing their partner to use their tool on a block
Directive-PutDown | [Block] | Directing partner to place the block on the ground
Directive-PutOn | Block | Directing partner to place block on anther block
Inquiry-Act | N/A | Asking what their partner is doing
Inquiry-Goal | [Block] | Inquiring about partner’s goal
Inquiry-NextStep | N/A | Asking what should be done next
Inquiry-OwnAct | [Block or Block-Pair] | Asking what they should do next
Inquiry-Possession | N/A | Asking about partner’s tools
Inquiry-Recipe | Block | Asking about a particular recipe node
Inquiry-Requirement | Block | Asking about required tools or blocks
Statement-Goal | Block | Stating their goal
Statement-Inability | N/A | Stating an inability
Statement-LackKnowledge | [Block] | Stating their lack of knowledge about a block
Statement-NextStep | Block or Block-Pair | Stating their next step
Statement-Other | N/A | Ambiguous statement
Statement-OwnAct | Block | Statement about the player’s own current act
Statement-Possession | Tool | Statement about the player’s own inventory
Statement-Recipe | Block and Block-Pair | Statement describing a step in the recipe
Statement-Requirement | Tool or Block or Pair | Statement about required tools or blocks for a step
Statement-StepDone | Block | Statement informing about completion of a step
BACKCHANNEL | N/A | off topic statements
OPINION | N/A | We should …; it must be …
AGREEMENT | N/A | sure; ok;
AnsAff | N/A | agreement to a question
CLOSING | N/A | bye; i’m out
AnsNeg | N/A | nope, i don’t have
ACKNOWLEDGMENT | N/A | right; done; indeed
AnsOth | N/A | I don’t know
OPENING | N/A | hi; hello
OrClause | N/A | … or …
APOLOGY | N/A | sorry
GameSpec | N/A | Specific to the game environment
other | N/A |

Table 6: The Dialogue move taxonomy. The information portraying dialogue acts
was expanded to give a more fine-grained description.


[a, b]Hiromasa Suzuki

# Observational constraints on the maximum energies of accelerated particles
in supernova remnants

Aya Bamba  Ryo Yamazaki  Yutaka Ohira

###### Abstract

Supernova remnants (SNRs) are thought to be the most plausible sources of
Galactic cosmic rays. One of the principal questions is whether they are
accelerating particles up to the maximum energy of Galactic cosmic rays
($\sim$PeV). In this paper, we summarize our recent studies on gamma-ray-
emitting SNRs. We first evaluated the reliability of SNR age estimates to
quantitatively discuss time dependence of their acceleration parameters. Then
we systematically modeled their gamma-ray spectra to constrain the
acceleration parameters. The current maximum energy estimates were found to be
well below PeV for most sources. The basic time dependence of the maximum
energy assuming the Sedov evolution
($\mathrel{\raisebox{3.01385pt}{\mbox{$\underset{\vbox{\hbox{$\sim$}}}{\vbox{\hbox{$\propto$}}}$}}}t^{-0.8\pm
0.2}$) cannot be explained with the simplest acceleration condition (Bohm
limit) and requires shock-ISM (interstellar medium) interaction. The inferred
maximum energies during lifetime averaged over the sample can be expressed as
$\lesssim 20$ TeV ($t_{{\rm M}}/\text{1~{}kyr})^{-0.8}$ with $t_{\rm M}$ being
the age at the maximum, which reaches $\sim$PeV only if $t_{\rm M}\lesssim 10$
yr. The maximum energies during lifetime are suggested to have a variety of
1–2 orders of magnitude from object to object on the other hand. This variety
will reflect the dependence on environments.

##  1 Introduction

Galactic cosmic rays are high-energy particles with an energy spectrum
approximated by a power-law and a maximum energy of $\approx 10^{15.5}$ eV
($\approx 3$ PeV). Their acceleration sites are still unclear despite of more
than 100 yr investigations. Supernova remnants (SNRs) are thought to be the
most promising acceleration sites that provide these high-energy particles.
According to analytical models for diffusive shock acceleration, these sources
are believed to actually supply particles with energies of $\lesssim$ PeV
(e.g., [1, 2]).

Gamma-ray observations have revealed that charged particles are accelerated to
energies above TeV (1012 eV) in young SNRs. Most of these SNRs feature,
however, spectral turnovers at energies below PeV. Recently, Tibet Air-Shower
array and LHAASO (Large High Altitude Air Shower Observatory) observations
have identified a number of PeV accelerators (PeVatrons) [3, 4], which
includes only a few SNRs. These results suggest that PeV particles accelerated
in early evolutionary phases have already escaped (e.g., [5, 6, 7, 8, 9, 10]),
or that there are almost no PeVatron SNRs. Other scenarios have also been
proposed in which SNRs in specific conditions become PeVatrons: e.g., very
young SNRs in dense environments [11].

In this paper, we summarize our recent observational works on investigation of
the maximum attainable energy of SNRs. In Section 2, we quantitate the
reliability of age estimates [12]. Section 3 summarizes our systematic
analysis on the gamma-ray spectra of 38 sources [13]. Using the age and gamma-
ray properties, we extract the basic time dependence and variety of the
maximum attainable energies in Section 4. Our basic conclusion is that SNRs
can be PeVatrons only if they accelerate particles very efficiently when they
are very young such as less than 10 yr (Section 5).

##  2 Reliability of supernova remnant age estimates

In order to discuss time dependence of particle-acceleration properties
quantitatively, we first evaluate the uncertainties in SNR age estimation
using systems with reliable age measurements $t_{\rm r}$, either the
historical age, light-echo age, $t_{\rm kin,ej}$ (kinematic age of free-moving
ejecta knots), or $t_{\rm kin,NS}$ (kinematic age of associated neutron star)
[12]. First, we measure $t_{\rm kin,NS}$ for available systems by estimating
the geometric centers of the SNR shells. The velocities of neutron stars are
taken from multi-epoch radio observations. Then we “calibrate” the general age
estimates, which are applicable for most cases111Dynamical age $t_{\rm dyn}$
and plasma age $t_{\rm p}$, by comparing them with the reliable estimates
$t_{\rm r}$.

The sample for this section consists of the SNRs with known historical or
light-echo ages, $t_{\rm kin,ej}$, or measurable $t_{\rm kin,NS}$. The best
age $t_{\rm b}$ is defined for each source, which is the most reliable
estimate. Details of the sample are explained in [12].

Ideally, where a symmetric supernova explosion occurs in a uniform ambient
density, the SNR will have a circular shell and so the geometric center can be
easily determined. In reality, however, the observed SNR shapes are not simply
circular. Thus, we model the projected morphology of the SNR shell with an
ellipse as a simple enough yet better alternative to a circle and estimate the
geometric center. Then we can calculate $t_{\rm kin,NS}$ using the neutron
star position and velocity.

Figure 1 plots $t_{\rm dyn}$ and $t_{\rm p}$ as a function of $t_{\rm r}$
(historical ages, light echo ages, and/or $t_{\rm kin,NS}$). Ideally, all the
points would be on the straight line on the $x$-$y$ plane ($y=x$). The $t_{\rm
dyn}$ and $t_{\rm p}$ values of most of the SNRs in our sample are found to be
in good agreement with their $t_{\rm r}$ within a factor of four. So, we
tentatively conclude that a systematic error of a factor of four is associated
with the general age estimates.

Figure 1: Comparison of a variety of age estimates for SNRs. The $x$-axis
represents reliable estimates $t_{\rm r}$ (circles for historical ages,
crosses for light-echo ages, inverted triangles for $t_{\rm kin,ej}$, squares
for $t_{\rm kin,NS}$, triangles for $t_{\rm kin,NS,prev}$). The $y$-axis
represents the dynamical (red) or plasma (blue) ages. For reference, the
linear functions $y=x,4x$, and $1/4\,x$ are also plotted.

##  3 Gamma-ray spectral properties

Here we describe the essence of our analysis and results performed in [13].
Details of the sample selection can be found in [13]. Figure 2 shows the
diameter $D$ and inferred shock velocity $v_{\rm ave}$ as a function of
$t_{\rm b}$.222We calculate this velocity assuming the Sedov model as $v_{\rm
ave}=D/5t_{\rm b}$. A representative Sedov model, for which a condition
$D=C_{D}t^{2/5}$ is assumed and the parameter $C_{D}$ is selected by eye to
roughly match the data, is also plotted in Figure 2. Both the $D$–$t_{\rm b}$
and $v_{\rm ave}$–$t_{\rm b}$ plots are close to the representative Sedov
model, indicating that the estimated age is plausible.

Figure 2:  Left and right panels show the SNR diameter $D$ and inferred
average shock velocity $v_{\rm ave}$ as a function of $t_{\rm b}$,
respectively. A condition $v_{\rm ave}=D/5t_{\rm b}$ is assumed. Note that the
objects shown here are only those with gamma-ray emissions. A representative
Sedov model, satisfying the condition $D=C_{D}t^{2/5}$ with the parameter
$C_{D}$ selected by eye to roughly match the data, is also plotted with black-
dashed lines.

In our work, hadronic emission is assumed to dominate the gamma rays, which
seems to be the case at least for several SNRs based on their spectral shapes
and energetics (e.g., [14]). The exponential cutoff energy ($E_{\rm cut}$) or
break energy ($E_{\rm br}$) of a gamma-ray spectrum reflect the maximum
energies of fresh accelerated particles [15, 9, 16]. If particles no longer
accelerated do not significantly contribute to the emission, an exponential-
like cutoff feature is expected. On the other hand, if the contribution of
escaping particles cannot be ignored, the gamma-ray spectrum will be similar
to a broken power-law. Since generally we cannot distinguish between these two
situations without detailed information on acceleration sites, the gamma-ray
spectra are fitted with both an exponential cutoff power-law model and a
broken power-law model. By fitting the Fermi, H.E.S.S., MAGIC, and VERITAS
spectra, we obtain parameters such as spectral index, cutoff, break, and
hardness ratio.

##  4 Time dependence of maximum acceleration energy and variety

Figures 3 (a) and (b) show $E_{\rm br}$ and hardness ratio333Flux ratio, 10
GeV–100 TeV over 1–10 GeV. over age, respectively. Both of them show general
decreasing trends with age with large scatters, which are consistent with
previous works [17, 18]. The $E_{\rm cut}$–$t_{\rm b}$ and $E_{\rm
br}$–$t_{\rm b}$ plots are modeled with a power-law model, and the best-fit
functions are obtained as $E_{\rm cut}=1.3_{-0.67}^{+1.1}~{}{\rm
TeV}~{}(t_{\rm b}/1~{}{\rm kyr})^{-0.81\pm 0.24}$ and $E_{\rm
br}=270_{-130}^{+240}~{}{\rm GeV}~{}(t_{\rm b}/1~{}{\rm kyr})^{-0.77\pm
0.23}$. Thus, based on our simple extrapolation with an assumption of the
Sedov evolution, only SNRs younger than $\sim 10$ yr have a chance to become
PeVatrons. This scenario matches some theoretical works [8, 11, 10]. The time
dependence of
$\mathrel{\raisebox{3.01385pt}{\mbox{$\underset{\vbox{\hbox{$\sim$}}}{\vbox{\hbox{$\propto$}}}$}}}t^{-0.8}$
is different from that expected in the simplest regime, Bohm limit
($t^{-0.2}$) [19], and requires wave damping via shock-ISM (interstellar
medium) collisions (e.g., [5, 20, 21, 16]444This discrepancy will be partly
due to the assumption of the Sedov evolution for all the sources.. We also
find a large variety of the maximum energy estimates even at the same ages: we
quantify this variety to be 1–2 orders of magnitude.

In the plot of the cutoff energy over hardness ratio in Figure 4, most sources
show a similar trend evolving from top right to bottom left, namely young
systems show a hard emission. There are some outliers, however, which can be
classified as “W28-like” and “RXJ1713-like” objects. The maximum energy
estimates are well constrained for these outliers because the contributions of
fresh accelerated particles and escaping particles can be separated well
(refer to [13] for details). Theoretical studies suggest that these peculiar
properties are due to complex gas environments [22, 9].

Figure 3: Plots of gamma-ray break energy (a) and hardness ratio (b) over age.
Grey regions represent the best-fit power-law functions and their $1\sigma$
confidence ranges.  Figure 4:  Scatter plot of gamma-ray cutoff energy over
hardness ratio (right) and two example gamma-ray spectra (left) extracted from
two outlier groups, W28-like and RXJ1713-like objects.

##  5 Summary

In our recent studies, we investigated the basic time dependence and variety
of acceleration properties of SNRs. We first calibrated general age estimates
using reliable estimates available only for specific systems. As a result, the
general estimates of the SNRs were found to agree with $t_{\rm r}$ within a
factor of four. A systematic analysis of 38 gamma-ray emitting SNRs using
their thermal X-ray and gamma-ray properties was performed. A spectral
modeling on their gamma-ray spectra has allowed us to constrain the particle-
acceleration parameters. Two candidates of the maximum energy of freshly
accelerated particles, the gamma-ray cutoff and break energies, were found to
be well below PeV for our sample.

The general time dependences of the maximum energy estimates for our sample,
$\mathrel{\raisebox{3.01385pt}{\mbox{$\underset{\vbox{\hbox{$\sim$}}}{\vbox{\hbox{$\propto$}}}$}}}t^{-0.8}$,
cannot be explained with the simplest acceleration condition of the Bohm
limit. The estimated average maximum energies of accelerated particles during
lifetime $\lesssim 20$ TeV $(t_{\rm M}/1~{}{\rm kyr})^{-0.8}$ are well below
PeV if the age at the maximum $t_{\rm M}$ is $\sim 100$–1000 yr. However, if
the maximum energy during lifetime is realized at younger ages such as $t_{\rm
M}<10$ yr, it can become higher and reach PeV. On the other hand, the maximum
energies during lifetime are suggested to have a large variety of 1–2 orders
of magnitude from object to object. This will reflect the dependence of the
acceleration processes on environments.

## References

  * [1] A. R. Bell.  MNRAS, 182:147–156, January 1978.
  * [2] P. O. Lagage and C. J. Cesarsky.  A&A, 125:249–257, September 1983.
  * [3] M. Amenomori, Y. W. Bao, X. J. Bi, et al., and The Tibet ASgamma Collaboration.  Nature Astronomy, 5(5):460–464, 2021.
  * [4] Z. Cao, F. A. Aharonian, Q. An, et al.  Nature, 594(7861):33–36, 2021.
  * [5] V. S. Ptuskin and V. N. Zirakashvili.  A&A, 403:1–10, May 2003.
  * [6] Y. Ohira, K. Murase, and R. Yamazaki.  A&A, 513:A17, April 2010.
  * [7] K. M. Schure and A. R. Bell.  MNRAS, 435(2):1174–1185, October 2013.
  * [8] M. Cardillo, E. Amato, and P. Blasi.  Astroparticle Physics, 69:1–10, September 2015.
  * [9] S. Celli, G. Morlino, S. Gabici, and F. A. Aharonian.  MNRAS, 490(3):4317–4333, December 2019.
  * [10] T. Inoue, A. Marcowith, G. Giacinti, A. J. van Marle, and S. Nishino.  ApJ, 922 (1):7, November 2021.
  * [11] P. Cristofari, P. Blasi, and E. Amato.  Astroparticle Physics, 123:102492, December 2020.
  * [12] H. Suzuki, A. Bamba, and S. Shibata.  ApJ, 914(2):103, June 2021.
  * [13] H. Suzuki, A. Bamba, R. Yamazaki, and Y. Ohira.  ApJ, 924(2):45, January 2022.
  * [14] M. Ackermann, M. Ajello, A. Allafort, et al.  Science, 339(6121):807–811, February 2013.
  * [15] Y. Ohira, K. Murase, and R. Yamazaki.  MNRAS, 410(3):1577–1582, January 2011.
  * [16] R. Brose, M. Pohl, I. Sushch, O. Petruk, and T. Kuzyo.  A&A, 634:A59, February 2020.
  * [17] H. Zeng, Y. Xin, and S. Liu.  ApJ, 874(1):50, March 2019.
  * [18] H. Suzuki, A. Bamba, R. Yamazaki, and Y. Ohira.  PASJ, 72(5):72, October 2020.
  * [19] J. Vink.  Physics and Evolution of Supernova Remnants.  Springer Charm,  2020\.
  * [20] V. S. Ptuskin and V. N. Zirakashvili.  A&A, 429:755–765, January 2005.
  * [21] H. Yasuda and S.-H. Lee.  ApJ, 876(1):27, May 2019.
  * [22] L. Nava and S. Gabici.  MNRAS, 429(2):1643–1651, February 2013.


# Towards Precision Aquaculture: A High Performance, Cost-effective IoT
approach

Rafael R. Teixeira, Juliana B. Puccinelli
Luis Poersch, Marcelo R. Pias and Vinícius M. Oliveira  FURG \- Rio Grande,
RS, Brazil
rafael, julianapuccinelli<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Ahmed Janati, Maxime Paris  BIOCEANOR
Sophia Antipolis, France
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Demand for ocean-based high-quality and sustainable fish protein soared in the
last decade. Unlike precision agriculture, aquaculture remains an under-
equipped farming activity. To fully realise a vision of precision aquaculture,
smart IoT can offer the essential tools to local producers worldwide. This
paper addresses the question of whether it is feasible to deploy a water
quality sensor network in aquaculture for real-time user visualisation and
decision making. The proposed system design and its implementation have been
validated in a typical aquaculture farming facility. Preliminary results
suggest the approach is feasible for short to medium LoRa communication range
(up to 110m from outdoor water tanks). Although sensor data loss occurred, the
end-to-end data path from the water sensors up to the user’s monitoring app
has been a positive outcome. The proposed IoT system approach provides the
ground for a sustainable precision aquaculture 111This technology-based work
has received funding from the European Union’s Horizon 2020 research and
innovation programme under grant agreement No 863034 (ASTRAL - All Atlantic
Ocean Sustainable, Profitable and Resilient Aquaculture - https://www.astral-
project.eu.

###### Index Terms:

Cost-effective IoT, Re-circulation Aquaculture System, LoRa network

##  I Introduction

Aquaculture is the farming of fish, crustaceans, molluscs, aquatic plants,
algae, and other organisms. Activities in aquaculture have seen a global
annual growth rate of 7%, highlighting their increased relevance to animal
protein production worldwide222FAO. 2018. The State of World Fisheries and
Aquaculture 2018 - Meeting the sustainable development goals. Public report.
As a result, the North and South Atlantic production has received considerable
government and policymakers attention from the ’90s onwards. For instance,
Brazil’s government stimulus and private investments helped increase the per
capita fish consumption in this country, thus leading to a net production of
707 thousand tons in 2015. Such tailored interventions have taken Brazil to
12th place in the world’s aquaculture league. On a global scale, aquaculture
fish production reached 82.1 million tons worldwide in 2018 according to an
FAO report [1]. The substantial social and economic impacts of aquaculture
farming cannot be underestimated as a catalyst for new employment posts
throughout the whole value chain. Such benefits reach the small producers
located on both sides of the Atlantic, including impoverished areas in low-to-
middle income countries (e.g. communities in South America and Africa). Given
the demand rise for high-quality animal protein to feed the world population,
marine aquaculture is under tremendous pressure to accelerate production in a
responsible, eco-friendly manner. A crucial enabler is the technology for
real-time monitoring of aquatic organisms in an operational setting. Rigorous
control of the environment’s physical-chemical parameters is the first step
towards an economically viable production. Unbalanced parameters can directly
affect species’ growth, generate pathogens and environmental conditions that
may hinder animal reproduction and healthy growth. Aquaculture is an
unpredictable biological system in which adverse outcomes cannot be ignored in
the management of a production system [2]. The European Commission adopted the
term Aquaculture 4.0 to contemplate the transfer of new technologies
investigated in other industries, including Internet of Things (IoT),
distributed computing, data analytics and artificial intelligence. The
Internet of Things (IoT) is a rapidly growing technological field, providing
social and economic benefits for all geographical areas but particularly low-
cost technology delivered to low-to-middle income countries. For example,
traditional water monitoring in aquaculture follows a very laborious and time-
consuming procedure whereby essential variables are measured. The high cost of
the needed human resources severely limits the monitoring and control of the
aquaculture facilities [3, 4].

IoT-driven monitoring and control in aquaculture is a challenging scenario in
itself because of a number of constraints:

  1. 1.

Site area: anything between 5 to 30 hectares;

  2. 2.

Difficult access (boats or off-road vehicles);

  3. 3.

Power supply: none in open-water; none to limited in land-based facilities;

  4. 4.

Communication coverage: limited to none;

  5. 5.

Climate and adverse weather conditions;

  6. 6.

Lack of reliable and suitable monitoring technology.

Aquaculture 4.0 offers tools that can be leveraged and applied to level up
production systems’ quality and effectiveness. Previous limitations can be
overcome when reliable technologies monitor and identify any undesired
condition or event of interest learnt from data. A key question to ask is the
following one: how feasible are the design and deployment of a water quality
sensor network that communicates data to an online platform for real-time
visualisation and user decision making? [5].

This paper presents initial validation results of the AquaGreen IoT system as
an attempt to shed some light on the solution space for the question above.
The specific contributions are:

  * •

System design specification and initial validation of cost-effective IoT
monitoring system (proof of concept) for real-world aquaculture farming.

  * •

In-loco validation of real-time, low-power long-range data communication based
on LoRaWAN.

This paper takes further Pister [6] smartdust vision to develop a bottom-up
approach for sensor application development. It departs, however, from the
biased view that sensor technology can be easily deployed into the target
environment (e.g. self-organising sensors ”thrown” from airplanes [6]). Unlike
this, the practicalities of placing IoT technology in the real world should be
fully appreciated for the smartdust vision to realise. The AquaGreen described
in this paper contributes to sustainable aquaculture in tackling a set of
specific U.N.’s Sustainable Development Goals333United Nations Sustainable
Development Goals: https://sdgs.un.org/goals: SDG 2 - Zero Hunger, SDG 9 -
Industry, Innovation, Infrastructure, SDG 11 - Sustainable and Connected
Communities around the Globe and SDG 14 - Life below Water.

###  I-A Water Quality Monitoring

High-quality water is a crucial requirement across the board for marine
aquaculture, where production performance is often excelled in those cases of
tight monitoring and control. Poor management of basic physical-chemical
parameters (temperature, dissolved oxygen, pH, turbidity and ammonia) can
result in significant losses leading to a production reduction of anything
between 20% and 40% [7]. A significant drop in dissolved oxygen levels, mainly
considering an intensive production, can lead to an undesirable loss in shrimp
farms. A few catastrophic animal losses have been experienced at FURG’s Marine
Aquaculture Facility (FURG-EMA), in some cases the slow user reaction (more
than 30 minutes) affected almost the entire cycle of shrimp production.
Temperature variation is one of the most critical factors to keep a tight
monitoring-control scheme. All the species’ physiological and behaviour
related activities, such as breathing, digestion, and feeding, are directly
linked to the water temperature levels. The higher the temperature, the
greater the level of animal activities. As a result, oxygen consumption is
increased likewise. Monitoring essential water quality variables is a massive
step towards aquaculture enhanced production. The term performance here takes
a broader view that puts side-by-side production and eco-friendly sustainable
processes.

###  I-B Aquaculture IoT Kits: User Requirements

The vision of IoT as advocated in [8] comprising ubiquitous embedded A.I.
sensors and cooperating objects [9] lead to a highly distributed network of
smart sensor devices communicating with both end-users and peer devices. Such
a vision has been presented to end-users in a workshop held in early 2021 at
FURG-EMA (Brazil). Further requirements were also solicited from the H2020
ASTRAL project consortium. Despite the users’ needs for new types of advanced
monitoring, say microplastic contaminants and animal biomass, the list that
follows capture design issues of IoT applied to aquaculture applications:

  * •

Essential water monitoring variables: temperature, dissolved oxygen, pH,
turbidity and ammonia;

  * •

Measurement sampling rate: one every second;

  * •

Real-time data communication: measurement data to reach the end-user with due
time for control intervention in case of detected anomalies;

  * •

Scalability: solution to function in sparse and remote located land-based
aquaculture farms;

  * •

Limited power availability: a power-efficient system for long-lasting data
gathering and communication.

This list calls for a system capable of monitoring essential variables even
when farmers are distant from the breeding and production sites. In addition,
the system enables intelligent real-time control and predictive anomaly
detection that maximises the producer awareness. A.I models can enhance the
knowledge of local producers, lower existing barriers of aquaculture
activities by the society at large.

###  I-C Recirculation Aquaculture System - RAS

Aquaculture is still an under-equipped sector in terms of intelligent systems.
However, regardless of size and scale, water quality remains the common ground
in farming facilities worldwide. RAS (Recirculation Aquaculture System) is a
method in which the production system is continuously treated and reused,
allowing the total or partial reuse of water, thus decreasing water waste in a
production cycle. In particular, the RAS approach develops a treatment scheme
system of the effluents generated in the cultivation through induced water
flow in the land-based tanks. In contrast, offshore marine aquaculture, farms
located at a distance from the coast, have experienced rapid growth due to
limited land resources and high competition for their use in some economic
sectors. In addition, offshore facilities employ specialised labour to achieve
robustness and profitability. Open sea farming offers sustainability more
inherently as strong ocean currents provide circulation and handling of waste.
However, long-term restrictions to offshore farming have made land-based RAS
popular in the last decade or so. Less exposure to diseases and environmental
threats are regarded as crucial benefits. A few effective industrial
processes, tailored training and sensor technology, are expected to bring
professionalism to producers. In addition, the use of water recirculation in
closed land-based systems (RAS) should become affordable for widespread sector
penetration.

##  II Related Work

Lafont (et al.) [7] argues that IoT applied to aquaculture is embryonic still.
This application has the potential to create a coherent development front
towards the concept of precision aquaculture. In that sense, the IoT
contribution goes beyond the networked wireless nodes. Sensor data analytics
play a key role in handling complex data-oriented challenges, including
anomaly detection, predictive facility operation and maintenance - to cite a
few. The authors in [10] discuss the easy-to-use aspects of IoT tools when
compared to the cumbersome tools that aquaculture producers commonly use in
day-to-day practices. The same work observes that such tools have to be
intelligent, accessible, easy to implement, reliable and highly efficient.
Both studies rely on cost-effective wireless sensors to put together an
infrastructure where easy-to-use UIs graphically present to end-users changes
in monitored water pH, dissolved oxygen, and other variables.

This paper goes beyond when it addresses the practical and subtle challenge of
a real-world pilot deployment. Specifically, this paper contributes with a
better understanding of the issues associated with sensor data communication
quality, communication range, and data volume in a small yet powerful real-
world deployment in a aquaculture facility. Possible control and actuation in
the target environment allow for a diverse set of smart interventions such as
adding stabilisation chemicals to the water.

##  III System Approach

IoT enhanced with artificial intelligence has potential for widespread use in
farming activities, particularly aquaculture. Nevertheless, researchers and
practitioners should overcome problems highly dependent on taking technology
to the target environment (e.g. limited power and communication coverage in
both land-based and offshore facilities). In conjunction with the
recirculation production system (RAS), sensors, and actuators, machine
learning will pave the way to the vision of smart precision aquaculture. FURG
is active in aquaculture research for new systems, methodologies and species.
FURG’s Marine Aquaculture Station (EMA) is a complete production facility
catering for the species: Whiteleg shrimp (Litopenaeus vannamei), Nile tilapia
(Oreochromis niloticus), Crassostrea tulipa (Crassostrea gasar), Sea lettuce
(Ulva S.P.) and Salicornia neei. The 2,800 sqm facility offers an integrated
multi-trophic system (IMTA) where two or more aquatic species from different
food chain levels are farmed together so that waste from one species can be
used as a resource to another, thus increasing circularity and reducing waste.
However, the facility provides limited power and cellular communication for
uplink internet connectivity. This scenario is representative of a traditional
land-based marine aquaculture fishery facility. This work focuses on the
features and limitations of a networked sensor system, its implementation and
preliminary prototype validation in the field.

Figure 1: Marine Aquaculture Station - FURG EMA

The first architecture version envisaged many unnecessary component-level
functionalities such as several data sources, edge data computing and in-node
versus cloud processing aspects. The second version, now simplified after
consultation with end-users, comprises two levels of data interaction, namely
local and external. The local interaction consists of sensor-actuator nodes
structured in a local communication layer and a gateway node capable of
gathering sensor data to be relayed to an external entity. In contrast, sensor
data is stored and streams processed in the external layer (e.g. servers in
the cloud). Authorised users access real-time analysed data for decision-
making, such as increasing water aeration in a specific tank). Figure 2
illustrates the proposed network architecture.

Figure 2: Simplified Network Architecture

###  III-A Local Layer

This layer addresses the communication from sensor nodes to the gateway over a
Low Power Wide Area Network (LPWAN). LoRa has been because of its small energy
consumption footprint and enhanced communication coverage [11]. Nodes sample
specialised water quality sensors. Collected sensor data are encoded in a LoRa
packet and transmitted to an in-range gateway node. Received sensor data
(encoded in a JSON template), are forwarded via HTTP to an external cloud
server.

###  III-B External Layer

Gateway nodes receive external connectivity either through an internet-
connected WiFi router or cellular network, whichever is present at the
aquaculture facility. In a few remote sites where there is limited to none
internet access, it is anticipated that the gateway node enters into a delay
tolerant network mode (DTN [12]). This particular DTN setup requires end-user
intervention to decide how to forward the stored data through some type of
device-to-device transfer or data off-loading via a smartphone cellular
interface. In this work, gateway nodes communicate sensor data to external
domain servers located in a commercial cloud infrastructure (e.g. AWS EC2
instances). A commercial ISP provides internet connectivity to the gateway
node. The external cloud server offers a customised sensor data API that
receives JSON formatted data and stores them into a NoSQL database for
improved scalability in volume handling. The server API also provides
endpoints for clients to consume the stored sensor data using a customised
scheme with a regular expression scheme for data searching.

###  III-C Embedded Hardware and Software

System extensibility is a prime design choice for sensor-enabled aquaculture
applications. The intention here is to efficiently extend the IoT Kit
functionality by supporting multiple communication protocols, analogue/digital
sensors and actuators, and onboard co-processing units. The embedded software
is the one that should do most of the work to load up software modules for new
types of integration. Also, system power management plays a role in optimising
battery charge. This led the team to choose an ESP32 board comprised of two
32-bit CPU cores and an ultra low power co-processor. ESP32 supports a deep
sleep mode that brings down the current draw to a level under 10µA. The SX1276
LoRa transceiver module creates a powerful sensor node capable of multi-
communication that gives flexibility for comparison and validation in a
variety of aquaculture farming facilities (e.g. from resourceful to limited
power and communication support). Also, the chosen node supports WiFi, BLE and
ESP-Now considered useful complementary communication technologies in
challenging IoT contexts. The idea is to provide functionality for edge
processing. For instance, anomaly detection algorithms for water quality data
can be embedded into the ESP32. The recently developed TinyML framework based
on Google’s TensorFlow Lite for
Microcontrollers444https://www.tensorflow.org/lite/microcontrollers offers a
new dimension for embedded machine learning in the proposed sensor node.

###  III-D Experimental Setup: Farm Radio Surveying

The experiments below were carried out to survey a typical aquaculture farm
(e.g. FURG-EMA):

  * •

Experiment 1: to validate the maximum range for LoRa communication within the
vicinity of the facility. The end-users carefully chose nine location points
used for the LoRa transmitter side. Sensor data packets were sent from each of
the locations towards a fixed positioned LoRa receiver (base station). Average
received signal strength (RSSI) was recorded.

  * •

Experiment 2: to provide a first-hand evaluation of the end-to-end application
data path (i.e. in-water sensing to a mobile app). The setup included
integrating a calibrated water temperature sensor attached to a sensor node, a
gateway node, a cloud server API and a mobile app. The latter consumes time-
series water temperature data to be overlaid in a customised user monitoring
app.

##  IV Initial Results

###  IV-A LoRa Range Issues (Experiment 1)

LoRaWAN is a widely adopted solution in IoT networks due to its capability to
connect to a wide area with many devices and still offer a low power
communication [13]. In this setup, LoRa is used as the physical layer protocol
of a LoRaWAN network. However, the low data transmission rates present a
challenge for IoT applications, especially those relying on image-based
sensors. In its low-power basic setup, Aquaculture IoT should rely on time-
series sensor data that is tiny enough to be transported through a LoRa-based
network. This experiment aims at determining whether LoRa communication can
provide sufficient network coverage in the field. A pair of TTGO LoRa32 v1.0
development boards were used to implement the sensor nodes. Each board
includes an ESP32 MCU, an SX1276 LoRa Transceiver, a 3dBi antenna and a
0.96inch OLED display. The system has a JST battery plug and a battery
charging unit. A Li-ion 3.7V battery powers up the sensor nodes.

Sensor nodes were programmed interchangeably as LoRa transmitters and
receivers. The chosen frequency of 915MHz complies with the local radio
regulations even though radio interference is unlikely typical deployment
sites. The system operated using SF7 spreading factor at a transmission power
of 17dB. Other system configurations could undoubtedly be used, but this one
provided the base for a field evaluation. The receiver node powered from a
reliable power source was placed at a distance from the fishery tank. The
transmitter nodes had their positions varied one at a time over the set of
nine possible locations. The transmitter sent a batch of 10 LoRa packets to
the receiver node. Received Signal Strength Indication (RSSI) values
associated with received packets were calculated based on the following
communication power loss model [14]:

$L_{fs}=32.45+20log(D)+20log(f)$

where $L{fs}$ is the free space loss in decibels, $D$ is the distance (in km)
between transmitter and receiver nodes, and $f$ is the frequency in MHz.

Figure 3 presents the testing locations explored at the aquaculture farm
(FURG-EMA). The locations were carefully selected to provide a more realistic
survey of the communication range and possible blind spots for sensor
deployment. Laser-based surveying and satellite images were also used to
calculate approximate linear distances between transmitter and receiver nodes.

Figure 3: Satellite view of measurement points

Points $A$, $B$ and $C$ are located under a greenhouse roof structure covering
some of the facility RAS water tanks (Figure 3). Points $D$, $E$ and $F$,
located in the southern limits of the perimeter, cover the farm’s external
tanks Additionally, northern location points ($I$, $H$ and $G$) provided
extended coverage for a complete site surveying. Results obtained from these
experiments (Table I) suggest that the selected LoRa module can cover the
communication between points located at the RAS greenhouse tanks and external
fishery water tanks. However, this initial configuration was unable to cover
the entire site area; for instance, the testing points $G$, $H$ and $I$ could
not be reached from the LoRa transmitter. A possible cause for this issue is
the actual distance of over 110m, several buildings in the middle (point $I$)
and a small tropical forest (points $G$ and $H$) as shown in Figure 3).
Experimenting with different antenna design and radio frequencies could
increase the transmitter’s range. IoT-based site surveying will not be an easy
task in real-world aquaculture farm deployments. Satellite-based images can
assist the survey task though. The issues that arise from the farm’s remote
location, either in offshore or in-land facilities, are far more challenging
to address as indicated in (Table I). Similarly to agricultural rural areas,
aquaculture farming requires a systematic approach for site radio surveying
that relies on fresh satellite images to guide a time-consuming yet needed
manual intervention.

TABLE I:
LoRa Range Test Results

Location | Linear Distance | Average RSSI
---|---|---
A | 43m | -108dBm
B | 55m | -110dBm
C | 70m | -109dBm
D | 104m | -109dBm
E | 102m | -110dBm
F | 56m | -107dBm
G | 143m | no signal
H | 117m | no signal
I | 120m | no signal

###  IV-B End-to-End Architecture Implementation (Experiment 2)

Putting aside the deployment challenges, the results obtained in Experiment 1
encouraged the research team and end-users to continue working towards a
preliminary implementation of a functional system. To fully implement and test
the proposed system architecture, the sensor node design included capabilities
for the adverse weather conditions of an open area environment. The functional
set of sensor nodes (Kits), embedded gateway, cloud server software and mobile
app comprise the end-to-end IoT Kit. This has been termed the AquaGreen
system. Local access restrictions to the aquaculture facility imposed in
February 2021 due to the worsening of the COVID-19 pandemic in Brazil limited
the planned experiment scope. Although the system network layer (Experiment 1)
has been assessed in normal operating conditions in December 2020, the
AquaGreen system had no other option but to be tested in an outside swimming
pool. This lab setup still offered rigorous conditions to validate technical
design assumptions.

A DS18b20 temperature sensor continuously monitors the pool’s water
temperature for a specific time window. Waterproof casing enclosed a few
system components, including the ESP32 sensor node board, 3.7 Li-ion battery
and cables used for the external radio antenna and attached sensor probe. The
development board had been attached to the internal wall through a strong
double-sided adhesive tape. Figure 4 shows a picture of the AquaGreen system.

(a) Internal view of the sensor node

(b) Sensor node encasing during the test

Figure 4: AquaGreen Sensor node prototype

To obtain the maximum battery life, the sensor node is kept in deep sleep mode
most of the time. During this period, the temperature sensor is also powered
down. On deep sleep mode, the overall system power consumption is reduced to
around 10µA. The chosen system duty cycle contemplates that every 10 minutes,
the sensor node wakes up from deep sleep to sample the sensors. The sample
data is encoded in a LoRa packet and sent to the gateway node. The LoRa packet
holds four key-value pairs of data samples (i.e. real-clock timestamp, sensor
node id, instantaneous water temperature sensor sample and battery voltage
level). Due to local regulations, only a maximum of 1% duty cycle is permitted
in LoRa communication. The system can send a packet (30 bytes payload plus 15
bytes header) every 10 seconds. To comply with the local rules, the actual
packet rate is one every 10 minutes. As the final setup step, the sensor node
was placed above the water in the swimming pool but having the temperate
sensor probe under the water in a 1.2 water depth approximately (Figure 4(b)).

The gateway node has been implemented with the same hardware used in range in
Experiment 1. The key difference lies on the embedded software, which now
implements the system architecture functional blocks. Specifically, in this
case, deep sleep mode in the gateway node was unnecessary due to the
availability of USB power throughout the experiments performed.

The gateway node acts as a LoRa receiver. Whenever a LoRa packet arrives from
the sensor node, the gateway unpacks it and encodes the data in a JSON file.
The node uses this document-oriented data format and an authentication token
to create a payload for an HTTP POST request to the cloud server. The gateway
node received Internet connectivity through the ESP32’s WiFi interface. The
real-time monitoring mobile app allows the user to authenticate into the
system, providing thus authorised access to the most recent sensor readings.
The cloud server functions as a data forwarding path to the end-users. It
securely stores the data received from the gateway nodes into a document-
oriented database. Then, the data is provided to a mobile app. Industry-grade
security has been used to secure the end-to-end data path with carefully
selected cloud-based services. The current server implementation uses
Javascript, node.js and MongoDB database. The server also handles user
authentication through JSON Web Tokens (JWT) to guarantee that only
authenticated entities are authorised to send and receive data from the cloud
server. Overall, the results of this simplified yet necessary experiment have
been positive. The AquaGreen system ran for a non-stop period of 66 hours up
to the point of fully discharging batteries in the sensor nodes. A total of
392 successful sensor readings were stored on the database during this time
interval. Unfortunately, a packet loss of 2.3% has been observed. Although
this has not impacted the application, it calls for further investigation into
whether a reliable communication protocol should be used.

##  V Conclusion

This work presented preliminary results of a real-world pilot in a challenging
environment for sensor and actuator networks. In addition, design principles
and best practices for IoT aquaculture application development were presented.
Experimental results shed some light on the feasibility of a water quality
sensor network that communicates data to an online platform for real-time
visualisation and user decision making. The first-hand results showed the
challenges for low-power long-range communication in a typical aquaculture
farm environment where buildings and tropical forests surround external tanks.
There are still many improvements to be made to the overall system.
Experimental with different radio and antenna configurations have the
potential to extend the communication range. A solar-based power source can
offer a viable solution to water tanks with no energy supply. As for the
decision-making software, analytical insights and predictions based on
previous time-series sensor recordings are under development. Predictive
monitoring and alerts to the users solve a significant issue: prompt user
reaction when water quality deteriorates. It only takes around 30 minutes for
a complete production loss in RAS-based tanks. Real-time processing is crucial
in this case. Although the training of neural networks remains a cloud-based
GPU task, TinyML and Tensorflow Lite for Microcontrollers provide a fast
inference engine that can be rightly embedded into the proposed system.

## References

  * [1] Why iot, big data & smart farming are the future of agriculture. [Online]. Available: http://www.fao.org/e-agriculture/news/why-iot-big-data-smart-farming-are-future-agriculture
  * [2] W. C. Valenti, H. P. Barros, P. Moraes-Valenti, G. W. Bueno, and R. O. Cavalli, “Aquaculture in brazil: past, present and future,” _Aquaculture Reports_ , vol. 19, p. 100611, 2021.
  * [3] M. Cordova-Rozas, J. Aucapuri-Lecarnaque, and P. Shiguihara-Juárez, “A cloud monitoring system for aquaculture using iot,” in _2019 IEEE Sciences and Humanities International Research Conference (SHIRCON)_ , 2019, pp. 1–4.
  * [4] K. S. S. Javvaji and M. A. Hussain, “Prototype of aquaculture using iot technologies,” in _2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT)_ , 2020, pp. 1–4.
  * [5] T. Abinaya, J. Ishwarya, and M. Maheswari, “A novel methodology for monitoring and controlling of water quality in aquaculture using internet of things (iot),” in _2019 International Conference on Computer Communication and Informatics (ICCCI)_ , 2019, pp. 1–4.
  * [6] J. M. Kahn, R. H. Katz, and K. S. J. Pister, “Emerging challenges: Mobile networking for “smart dust”,” _Journal of Communications and Networks_ , vol. 2, no. 3, pp. 188–196, 2000.
  * [7] M. Lafont, S. Dupont, P. Cousin, A. Vallauri, and C. Dupont, “Back to the future: Iot to improve aquaculture : Real-time monitoring and algorithmic prediction of water parameters for aquaculture needs,” in _2019 Global IoT Summit (GIoTS)_ , 2019, pp. 1–6.
  * [8] F. Xia, L. T. Yang, L. Wang, and A. Vinel, “Internet of things,” _International journal of communication systems_ , vol. 25, no. 9, p. 1101, 2012.
  * [9] M. B. P. J. M. A. O. A. Wolisz, _Cooperating Embedded Systems and Wireless Sensor Networks_. Willey, 2008.
  * [10] C. Dupont, P. Cousin, and S. Dupont, “Iot for aquaculture 4.0 smart and easy-to-deploy real-time water monitoring with iot,” in _2018 Global Internet of Things Summit (GIoTS)_ , 2018, pp. 1–5.
  * [11] R. S. Sinha, Y. Wei, and S.-H. Hwang, “A survey on lpwa technology: Lora and nb-iot,” _ICT Express_ , vol. 3, no. 1, pp. 14–21, 2017.
  * [12] K. Fall, “A delay-tolerant network architecture for challenged internets,” in _Proceedings of the 2003 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications_ , ser. SIGCOMM ’03. New York, NY, USA: Association for Computing Machinery, 2003, p. 27–34.
  * [13] T. Attia, M. Heusse, B. Tourancheau, and A. Duda, “Experimental characterization of lorawan link quality,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , 2019, pp. 1–6.
  * [14] Lora documentation - free space losses. [Online]. Available: https://lora.readthedocs.io/en/latest/#free-space-losses