NAOInstitute, The University of Auckland1 Department of Ecoscience, Aarhus
University2 Wellington Zoo3

# Behaviour Modelling of Social Animals via Causal Structure Discovery and
Graph Neural Networks

Gaël Gendron1 Yang Chen1 Mitchell Rogers1 Yiping Liu1 Mihailo Azhar2 Shahrokh
Heidari1 David Arturo Soriano Valdez1 Kobe Knowles1 Padriac O’Leary1 Simon
Eyre3 Michael Witbrock1 Gillian Dobbie1 Jiamou Liu1 Patrice Delmas1

###### Abstract.

Better understanding the natural world is a crucial task with a wide range of
applications. In environments with close proximity between humans and animals,
such as zoos, it is essential to better understand the causes behind animal
behaviour and what interventions are responsible for changes in their
behaviours. This can help to predict unusual behaviours, mitigate detrimental
effects and increase the well-being of animals. There has been work on
modelling the dynamics behind swarms of birds and insects but the complex
social behaviours of mammalian groups remain less explored. In this work, we
propose a method to build behavioural models using causal structure discovery
and graph neural networks for time series. We apply this method to a mob of
meerkats in a zoo environment and study its ability to predict future actions
and model the behaviour distribution at an individual-level and at a group
level. We show that our method can match and outperform standard deep learning
architectures and generate more realistic data, while using fewer parameters
and providing increased interpretability.

11footnotetext: Corresponding author. Email address:
<EMAIL_ADDRESS>
##  1\. Introduction

(a) Carrying a pup.

(b) Digging.

(c) Low sitting or standing.

(d) Interacting with a foreign object.

(e) Playfighting.

(f) Raised guarding.

Figure 1. Examples of meerkat behaviours in the Meerkat Behaviour Recognition
Dataset (Rogers et al., 2023).

Understanding and modelling non-human animal behaviour is a fundamental task
in ecological research, with wide applications ranging from unusual behaviour
detection (Buskirk et al., 1981), population dynamics (Morales et al., 2010),
habitat selection analysis (Van Moorter et al., 2016), and disease spread
modelling (Dougherty et al., 2018). One particular application of behaviour
modelling is to monitor the well-being of animals in zoo environments, where
evidence-based approaches for animal behaviour monitoring and prediction are
essential to help zoos maximise the well-being of their captive wild animals
(Rose and Riley, 2021; Scott, 2014; Scott et al., 2017).

To model animal behaviour, classical rule-based methods rely predominantly on
the manual identification of correlations. In many cases, these methods begin
with a hypothesis and subsequently validating it using the collected data (Van
Moorter et al., 2016; Buskirk et al., 1981). However, despite the achievements
of these approaches, they rely on human expertise, which can be expensive and
impractical when dealing with complex behavioural rules. The advent of machine
learning has opened up the possibility of simultaneously considering multiple
factors when investigating complex correlations (Valletta et al., 2017).
However, learning correlations without recovering the cause and effect
knowledge cannot provide a full understanding of the studied phenomenon
(Schölkopf et al., 2021). While existing research has largely concentrated on
the interplay between behaviour and environmental factors, the cause-and-
effect relationships among behaviours have been less explored. Causal
relationships among behaviours are particularly prominent in social mammals,
such as meerkats and chimpanzees (Clutton-Brock et al., 2004; Hinde, 1976;
Wakefield, 2013) and are often interrelated in complex ways. In zoo
populations, human intervention adds an additional layer of complexity.
Consequently, determining the causal relationships between behaviours that
evolve over time can be challenging.

Causality theory for time series aims to recover the causal dependencies
between variables that evolve over time (Moraffah et al., 2021; Gendron et
al., 2023). These methods are well suited to discriminate true causal links
from spurious correlations (e.g. variables can be correlated because they
share a common cause but have no causal links between them). A small body of
work recovers the causal structure of animal behaviours but focuses on insect
swarms (Lord et al., 2016) or bird flocks (Chen et al., 2020) and does not
attempt to model the complex social interactions of individuals.

In this work, we propose an approach based on Causal Structure Discovery
(Gendron et al., 2023) and Neural-Causal Inference (Zecevic et al., 2021) to
(1) automatically discover the causal relationships between the behaviours of
individuals in a social group and (2) model and predict the behaviours of
individuals over time. We build a causal graph from the data using a Causal
Structure Discovery algorithm to tackle the first objective. We present the
algorithm in Section 3.1. We then use the causal graph to constrain a Graph
Neural Network (Scarselli et al., 2009; Kipf and Welling, 2017) to perform
Causal Inference and predict the next behaviour of an individual, as described
in Section 3.2. In Section 4, we apply the proposed method to simulate the
behaviours of a mob of meerkats in an enclosure of the Wellington Zoo. Figure
1 illustrates some examples of the observed behaviours. We study the
prediction abilities of the model and compare it with deep learning baselines.
Then, we simulate the behaviours at both the individual and group levels,
comparing the results to the ground truth. We also discuss the advantages of
the proposed method in terms of explainability. Our code is available at:
https://github.com/Strong-AI-Lab/behavior-causal-discovery. The contributions
of this study are as follows:

  * •

We propose a novel interpretable method to model and predict the behaviours of
social animals using causal structure discovery and graph neural networks.

  * •

We apply our method to model a group of meerkats and show that it competes
with standard deep learning models on behaviour prediction tasks,
outperforming them in low temporal context settings.

  * •

We simulate a group of meerkats over time and show that our method yields more
realistic data than deep learning models.

##  2\. Related Work

#### Behaviour Modelling

To model animal behaviour, rule-based methods initially propose hypotheses and
then validate them using collected data, as in Van Moorter et al. (2016) and
Buskirk et al. (1981). Timberlake (1997) emphasises that these approaches can
discover useful causal laws for animal behaviours but these laws can be hard
to transfer to real-world situations and may carry anthropomorphic biases.
Sumpter et al. (2012) furthermore distinguishes local and global hypotheses,
i.e. hypotheses about the behaviour of a single individual against the
macroscopic behaviour of a group. Approaches from neuroscience study animal
behaviours from a cognitive perspective, but they cannot be applied using
observational data alone Shettleworth (2001, 2009). Automatically modelling
behaviours from data with little inductive bias can mitigate these challenges.
Machine learning has emerged as a powerful tool for investigating complex
rules by considering numerous factors (Valletta et al., 2017). For instance,
Shamoun-Baranes et al. (2012) use decision trees to predict the behaviour of
oyster-catchers from sensory data. Rew et al. (2019) proposed an LSTM model to
predict animal migration routes while considering weather and terrain. Methods
based on causality theory have also gained prominence for extracting cause-
and-effect relationships from interdependent animal behaviours (Porfiri, 2018;
Lord et al., 2016; Chen et al., 2020). However, the existing work has mostly
focused on easing hypothesis generation or predicting specific features, and
not on building a behavioural model that could simulate data.

#### Causality

Causality provides a theoretical framework for recovering and estimating cause
and effect variables from data observations. Two popular approaches are the
Potential Outcome Framework (POF) (Rubin, 1974; Splawa-Neyman et al., 1990;
Yao et al., 2021) and the Structural Causal Model (SCM) (Pearl, 2009a, 2014;
Xia et al., 2021). The Potential Outcome Framework usually models dependencies
between only two variables. SCMs, however, represent causal dependencies
between multiple variables as a graphical model, where the variables are the
nodes and the dependencies are the edges; this framework is well suited to our
work. Construction of causal graphs can be divided into two tasks: Causal
Structure Discovery and Causal Inference (Gendron et al., 2023). The former
recovers the structure of the graph, whereas the latter estimates the
functions linking the effect variables to their causes. Standard Causal
Structure Discovery methods are based on statistical conditional independence
tests between variables. The most noted algorithm is PC (Spirtes and Glymour,
1991; Spirtes et al., 2000), which can recover the complete causal graph if
all causal variables are observed (i.e. there are no hidden causes that can
act as confounding effects). Fast Causal Inference (FCI) (Spirtes et al.,
2000) and Real Fast Causal Inference (RFCI) (Colombo et al., 2012) extend PC
to handle confounding effects. Time-series FCI (TsFCI) (Entner and Hoyer,
2010) adapts FCI to time-series data; variables of interest are restricted to
a sliding temporal window, allowing the algorithm to see only the variables at
the current and previous $\tau$ timesteps. PCMCI (Runge et al., 2019a)
combines PC with Momentary Conditional Independence (MCI) tests to prune the
set of variables of interest before building dependency links, allowing
handling of large-scale data. Once the causal structure is known, Causal
Inference methods can be used to estimate the links between variables.
Traditional approaches include do-calculus (Pearl, 2009b), a Bayesian
inference framework, and regression methods such as Classification and
Regression Trees (CART) Loh (2011). More recent Causal Inference methods
called Neural-Causal models rely on deep neural networks, in particular Graph
Neural Networks (Zecevic et al., 2021), to estimate causal dependency
functions (Xia et al., 2021).

#### Graph Neural Networks

Graph Neural Networks (GNNs) are a family of deep learning models that use a
graph as an input and apply operations to update the node values (Scarselli et
al., 2009; Kipf and Welling, 2017). The nodes in the graph contain feature
vectors, and GNNs aggregate information from neighbouring nodes (nodes
connected by an edge) to update these feature vectors. By applying multiple
GNN operations, nodes can be updated based on information from distant nodes
in the graph. Popular GNN architectures include Graph Convolution Networks
(GCN) (Kipf and Welling, 2017), which apply convolution filters to graphs,
Graph Attention Networks (GAT) (Velickovic et al., 2018), which use attention
mechanisms to weigh the influence of neighbours on one node, and GATv2 (Brody
et al., 2022) which modifies this GAT architecture to increase the
expressivity of the attention weights.

##  3\. Causal Behaviour Modelling

Figure 2. Causal Modelling Framework. The model is composed of two components
fed from tabular data. The array contains sequences of behaviours from the
studied individuals (on the left, in gray). The Causal Structure Discovery
module generates a causal graph that links the behaviours together (on top, in
red). The graph contains naive transition probabilities between behaviours.
These probabilities are learned by the Neural-Causal model (at the bottom, in
green). The network is constrained by the causal graph to follow causal paths.
The resulting model combines both outputs (on the right, in red).

Causal Modelling Framework. The model is composed of two components fed from
tabular data. The array contains sequences of behaviours from the studied
individuals (on the left, in gray). The Causal Structure Discovery module
generates a causal graph that links the behaviours together (on top, in red).
The graph contains naive transition probabilities between behaviours. The
complex transition functions are learned by the Neural-Causal model (at the
bottom, in green). The network is constrained by the causal graph to follow
causal paths. The resulting model combines both outputs (on the right, in
red).

Modelling the behaviour of individuals from real-world data, particularly
social animals, is a challenging task. Real-world data from uncontrolled
environments may not contain the full scope of causes influencing individuals’
behaviour. Some uncontrolled causes may not be identifiable from available
data. Moreover, the data can contain biases towards representing some
behaviours more than others if the latter occur only rarely. Data scarcity
also hinders the data-dependant training of deep neural networks. In
particular, the data imbalance towards common behaviours makes it challenging
to learn from i.i.d. data. Finally, the sequences of behaviours being highly
stochastic, the absence of a known true policy complicates the evaluation of a
proposed model against ground truth.

To address these challenges, we propose an approach based on Causal Structure
Discovery and Causal Inference, shown in Figure 2. We build a causal model
from time-series with categorical data containing the states (i.e. behaviours)
to be learned and contextual information. The causal model contains the
transition functions between the states, and causal dependencies between the
node variables (what nodes are causes of the transition from one state to the
next). The method is divided into two modules: a Causal Structure Discovery
module that recovers the dependencies and structure of the causal graph, and a
Causal Inference module that learns the transition functions given the data
and graph structure. Sections 3.1 and 3.2 describe the architectures of the
two modules.

###  3.1. Causal Discovery of Behavioural Structure

We aim to build a causal graph that models the links between the behaviours of
social animals. We chose to represent a single individual or agent and its
immediate surroundings. To simulate a multi-agent community, multiple causal
models can be run concurrently. In our current work, for simplicity, we assume
that all agents share the same causal processes.

The variables of interest in the graph for a timestep $t$ are the values of
the possible states $S_{t}$ that the agent can be in (i.e. the behaviours,
illustrated in Figure 1) and the context information $C_{t}$: the location of
the agent, the presence of neighbours in a particular location, and the
presence of neighbours doing a particular behaviour. All variables are binary
and represent the presence or absence of a feature of interest. The set of
variables at a given timestep is
$\mathbf{X_{t}}=(X^{1}_{t},\dots,X^{N+M}_{t})=(S^{1}_{t},\dots,S^{N}_{t},C^{1}_{t},\dots,C^{M}_{t})$,
where $N$ is the number of possible states or behaviours and $M$ is the number
of context variables. We emphasise that during inference, only a subset of the
learned graph is used, because only one state $S_{t}$ is activated and a
subset of $C_{t}$ is activated. The causal model aims to recover the cause-
effects mechanisms that lead an agent to make the decision to change (or
maintain) its behaviour, i.e. the links $X^{i}_{t-k}\rightarrow X^{j}_{t+1}$.
$0<k<\tau$, where $\tau$ is the maximum size of the past time window
considered, i.e. the maximum time in the past we can look back. As the context
variables depend on causes not available to the system, we only aim to find a
subset of links $X^{i}_{t-k}\rightarrow S^{j}_{t+1}$. The task can also be
described as finding the parents $\mathcal{P}(S^{j}_{t+1})$ of each variable
$S^{j}_{t+1}\in\\{S^{i}_{t+1}\\}_{i\in 1,\dots,N}$.

We use the PCMCI algorithm (Runge et al., 2019b) to discover the causal
structure of the behaviour model. The algorithm works in two steps:

  * •

The first step consists in using the PC algorithm (Spirtes and Glymour, 1991;
Spirtes et al., 2000) to prune the set of possible parent variables to a small
subset of relevant parents $\hat{\mathcal{P}}(S^{j}_{t+1})$.

  * •

The PC algorithm is not built for time-series and may contain false positives.
To eliminate these, the second step uses a Momentary Conditional Independance
(MCI) test to determine whether a causal link exists between the potential
parent and child variable. The MCI test is defined in Equation 1.

$X^{i}_{t-k}\not\\!\perp\\!\\!\\!\perp
S^{j}_{t+1}|\hat{\mathcal{P}}(S^{j}_{t+1})\setminus\\{X^{i}_{t-k}\\},\hat{\mathcal{P}}(X^{i}_{t-k})$
(1)

The MCI test verifies whether variable $S^{j}_{t+1}$ is independent of
$X^{i}_{t-k}$ conditioned on the parents of $S^{j}_{t+1}$ (except
$X^{i}_{t-k}$) and the parents of $X^{i}_{t-k}$. Observing the parents of the
two variables block correlation effects. If the two variables are not
independent, then $X^{i}_{t-k}$ must be a cause of $S^{j}_{t+1}$. If
$S^{j}_{t+1}$ or $X^{i}_{t-k}$ share unobserved parent variables, they can act
as confounding effects and the true causal graph may not be recoverable. An
example of recovered structure is given in Section 4.6.

###  3.2. Behaviour Inference

The aim of the Causal Inference module is to estimate the function
$S^{j}_{t+1}=f_{j}(\mathcal{P}(S^{j}_{t+1}),\epsilon^{j}_{t+1})$ for every
variable $S^{j}_{t+1}\in\\{S^{i}_{t+1}\\}_{i\in 1,\dots,N}$ where
$\mathcal{P}$ denotes the parents of $S^{j}_{t+1}$,
$\mathcal{P}(S^{j}_{t+1})\subset\\{\mathbf{X_{t}},\mathbf{X_{t-1}},\dots,\mathbf{X_{t-\tau}}\\}$,
and $\epsilon^{j}_{\mathbf{X_{t-1}}}$ is random noise representing exogenous
mechanisms.

We use a Graph Neural Network for the Causal Inference module to take
advantage of the causal structure generated by the Causal Structure Discovery
module. The choice of a GNN is motivated by its ability to represent causal
mechanisms under the Structural Causal Model framework (Xia et al., 2021;
Zecevic et al., 2021). Figure 3 shows an overview of the module architecture.

Figure 3. Neural-Causal Model based on Graph Neural Networks. The causal graph
is built using the Causal Structure Discovery module (on top, in red) and
provided to the GNN (at the bottom, in green). The input values are provided
as node features. The GNN aggregates the features following the causal
dependencies and generates a probability vector for the next timestep using a
linear layer (on the right, in blue).

First, the input is provided to the Causal Structure Discovery module, which
generates the structure of the causal model. The graph can exhibit
dependencies up to $\tau$ timesteps in the past. The corresponding adjacency
matrix is provided to the GNN. Input variables are formatted to be ingested by
the GNN. Each binary feature of interest is converted to a one-hot encoded
vector, and represents the features of a node in the graph. Because all values
in the graph up to $t$ are in the data, the GNN must only compute the
aggregation function for the variables at timestep $t+1$. Therefore, only a
fraction of the graph is required for inference. To obtain a comprehensive
overview of the model abilities, we also trained intermediate GNNs tasked with
computing the intermediate values (e.g. computing
$S^{j}_{t-\tau+1}=f_{j}(\mathcal{P}(S^{j}_{t-\tau+1}),\epsilon^{j}_{t-\tau+1})$).
We use a final linear layer to convert the feature vectors into probabilities
of variable presence. The general aggregation function of the GNN for our
settings is given by Equation 2.

$S^{j}_{t+1}=\phi_{\mathbf{\theta_{2}}}\left(S^{j}_{t},\bigoplus_{x\in\mathcal{P}(S^{j}_{t+1})}\gamma_{\mathbf{\theta_{1}}}(S^{j}_{t},x)\right)$
(2)

Vector $S^{j}_{t+1}$ represents the features of state $j$ at timestep $t+1$,
$\mathcal{P}(S^{j}_{t+1})$ is the set of parents of $S^{j}_{t+1}$ and
$\bigoplus$ is a generic aggregation operator (e.g. sum $\sum$ or product
$\prod$). GNN layers typically include the previous value of the variable of
interest $S^{j}_{t}$. $\gamma$ and $\phi$ are functions parameterised by
$\theta=\\{\theta_{1},\theta_{2}\\}$.

##  4\. Application to Behaviour Prediction

In this section, we apply our method to the problem of modelling the behaviour
of social animals (meerkats). We simulate individual behaviours using the
proposed method. We first measure the performance of the model for direct
prediction. Then, we model the behaviours of a single individual within a
group, with the others evolving according to the data, and investigate how
accurate the generated series is, compared to the ground truth. Then, we model
the entire community and study how the group behaves as a whole. Finally, we
discuss the interpretability of the model.

###  4.1. Dataset and Data Preprocessing

We apply our model to the Meerkat Behaviour Recognition Dataset (Rogers et
al., 2023), a collection of 20 minutes annotated videos of a mob of meerkats
in the Wellington Zoo from static cameras. The dataset tracks the position and
behaviour of individual animals through video frames, and the distances
between individuals is determined using an estimation of their 3D positions.
Figure 2 illustrates the format of the data: for every timestep, the behaviour
and location of each individual is classified within pre-defined categories.
Nearby and distant meerkats are indicated, based on the computed distances
between individuals.

As mentioned in Section 3, modelling animal behaviour using observational data
is challenging. The individuals tracked in the Meerkat Behaviour Recognition
Dataset express few behaviours and can maintain the same behaviour for a long
period of time (for several minutes). The majority of the dataset contains
sequences with little updates to the variables. The remaining sequences can be
of high intensity, with behaviours changing rapidly (in less than a few
seconds). Therefore, the amount of diverse data is relatively small and not
independent and identically distributed (non-i.i.d.). To address this issue,
sequential duplicate information was removed from the dataset. For a given
timestep, we remove all the following timesteps that show no evolution in the
state of the system. This removal aims to balance the dataset and remove
biases towards majority behaviours. However, obtaining fully i.i.d. data is
not feasible because of the very low occurrence of some of the behaviours, as
shown in Figure 4. We address this constraint by using our proposed causal
model.

allogroomcarry pupdig burrowforaginggroomhigh sitting/standing (vigilant)human
interactioninteract with pupinteracting with foreign objectlow
sitting/standing (stationary)lying/resting (stationary)movingplayfightraised
guarding (vigilant)sunbathe$0$$0.1$$0.2$$0.3$FrequencyTrain dataTest data
Figure 4. Distribution of the behaviours in the training and test data.

###  4.2. Experimental Setup

Our experiments aim to estimate how well our model can predict the next state
or behaviour $s_{t+1}$ of an individual (i.e. the selected state among the
$S^{j}_{t+1}\in\\{S^{i}_{t+1}\\}_{i\in 1,\dots,N}$ random variables), given
its present and past states $s_{t}^{*},s_{t-1}^{*},\dots,s_{t-\tau}^{*}$ and
the context information $C_{t}^{*},C_{t-1}^{*},\dots,C_{t-\tau}^{*}$. Here, ∗
indicates the results of the ground truth policy. To study the impact of the
context window size, we also ask the model to recover the intermediate values,
i.e. $s_{t}^{*}$ given $s_{t-1}^{*},\dots,s_{t-\tau}^{*}$, and $s_{t-1}^{*}$
given $s_{t-2}^{*},\dots,s_{t-\tau}^{*}$, until $s_{t-\tau+1}^{*}$ given
$s_{t-\tau}^{*}$. The behaviour of meerkats is stochastic, and we must use
measures that take this stochasticity into account.

We use a value of $\tau=5$. We select this value as a trade-off between the
amount of information carried and the complexity of the causal discovery
required. We perform our experiments using several GNN architectures: GCN
(Kipf and Welling, 2017), GAT (Velickovic et al., 2018) and GATv2 (Brody et
al., 2022). We use a single GNN layer. Adding more layers would create non-
causal paths in the graph. As the GNN layers aggregate information from past
layers, stacking layers provides nodes with information from a greater
distance in the past. This information may not be blocked by the intermediate
node, which can create a non-causal backdoor path Zhang et al. (2022).

We compare our model against two baselines: a Long Short-Term Memory (LSTM)
model (Hochreiter and Schmidhuber, 1997) and a Transformer model (Vaswani et
al., 2017). For a fair comparison, the LSTM has one layer and a hidden size of
128, similar to the GNNs. We use the original encoder-decoder Transformer
architecture with three heads, one encoder, and one decoder layer to keep the
model size as close as possible to the GNN sizes. The size of each model is
listed in Table 1. The GNN models have smaller sizes for inference (with
$\delta\tau$ window) as we use a different GNN to compute each intermediate
value. Therefore, only one of the trained GNNs is required for inference (see
Section 3.2). The LSTM and Transformer architectures perform the intermediate
computations by default, so the same models are used for inference. As a
result, all baseline models are larger than the proposed models. We also build
ablated models for comparison.The first model is based only on the Causal
Structure Discovery module. We convert the measures of the causal strength
between the variables into probabilities using softmax aggregation. The second
family of ablated models contains both Causal Structure Discovery and Causal
Inference modules but uses a reduced graph
$\mathcal{G_{\overline{\text{corr}}}}$ instead of the full graph
$\mathcal{G}$. $\mathcal{G_{\overline{\text{corr}}}}$ does not contain links
in which the causal strength measures are below a statistical significance
threshold. We compare the performance of the ablated models against that of
the full models and baselines. All models are trained for 10 epochs with their
default hyperparameters.

Table 1. Number of parameters of each studied model. PCMCI is non-parametric and is omitted from the table. The first row corresponds to the total number of trained parameters, and the second row corresponds to the number of parameters used to compute next step prediction with a history of size $\tau$.  | GCN | GAT | GATv2 | LSTM | Trans.
---|---|---|---|---|---
Param. | 108.9K | 109.8K | 208.8K | 113.7K | 693.7K
Param. $\delta\tau$ | 26.4K | 26.6K | 43.1K | 113.7K | 693.7K

###  4.3. Behaviour Prediction

In this section, we model the behaviour of a unique individual over time. We
predict a sequence of behaviours $s_{t+1},s_{t+2},\dots$ from some initial
states $s_{t}^{*},s_{t-1}^{*},\dots,s_{t-\tau}^{*}$ and context
$C_{t}^{*},C_{t-1}^{*},\dots,C_{t-\tau}^{*}$. The context remains unchanged
for future predictions. E.g. $C_{t+1}^{*},C_{t+2}^{*}$ are used to predict
$s_{t+1},s_{t+2}$. In other words, the prediction $s_{t+1}$ does not affect
the context information. The other meerkats behave as expected in the ground
truth.

The behaviour of meerkats is stochastic. If trained on the available data
without access to the inner mechanisms of a meerkat, even a perfect policy
could not reproduce the ground truth data. Therefore, we do not attempt to
measure the long-term accuracy between the ground truth and the data generated
by the model. We study the difference between
$s_{t}=f(s_{t_{1}},\dots,s_{t-\tau},C_{t-1},\dots,C_{t-\tau})$ and
$s_{t}^{*}$. We report the accuracy measure on this task as a first indicator.
However, as multiple behaviours can be considered valid answers for a given
timestep, we also measure the Mutual Information ($I$) (Shannon, 1948; Kreer,
1957) between the predicted and expected policy. The Mutual Information $I$ of
two random variables corresponds to the amount of information we can obtain
about one variable by observing the other, and is defined by the following
equation:

$I(S_{t};S_{t}^{*})=H(S_{t})-H(S_{t}^{*}|S_{t})$ (3)

$H$ is the Shannon’s entropy of a random variable. We consider the Mutual
Information between the random variable representing the behaviour prediction
for the current step $S_{t}$ and the random variable representing the true
behaviour $S_{t}^{*}$. The Mutual Information returns a value between 0 and 1.
Intuitively, it corresponds to the amount of uncertainty of $S_{t}^{*}$ that
is removed by knowing $S_{t}$.

Table 2 shows the results of the experiments. The first Acc. column includes
the computation of the intermediate values. The full-size model is used for
these computations. The Acc. $\delta\tau$ and Mutual I. metrics use the final
output only and only require the smaller version of the model. Unsurprisingly,
the PCMCI performs consistently worse than all models on every metric. For the
accuracy metric on the full prediction, PCMCI+GCNG performs the best.
PCMCI+GCN${}_{\mathcal{G_{\overline{\text{corr}}}}}$ achieves similar
accuracy, within the standard deviation interval. Overall, the models with the
ablated graph perform similarly to their counterparts with the full graph.
This highlights that the causal paths contain the necessary information for
the task, and removing the correlation paths does not significantly impact
performance. PCMCI+GATv2G also performs well, achieving an accuracy similar to
that of the LSTM. The Transformer model achieves a lower accuracy than all the
other models, with the exception of PCMCI. As Transformers often require a
large quantity of data to learn, data scarcity is a possible reason for this
poor performance. Interestingly, the GNN models obtain greater accuracy on the
intermediate task, with lower temporal context. In the final prediction task
(with $\delta\tau$ context), the LSTM achieves the highest accuracy.
PCMCI+GCNG was the second-best model. The LSTM also achieves the highest
Mutual Information, although the scores of all the models are very low.
PCMCI+GCNG and the Transformer achieves similar $I$ scores.

The models can approximate the true behaviour of an individual agent in more
than half of the cases but the low Mutual Information scores show that the
models contain little information regarding the causal mechanisms of the
agents. We also observe that the proposed Neural-Causal models can yield
similar or better performance than well-established models for a significantly
smaller size network.

Table 2. Performance of the models on the individual-level prediction tasks. The mean and standard deviation across 3 training runs is shown. Acc. is the accuracy for the intermediate and final timesteps. Acc. $\delta\tau$ is the accuracy for the last prediction only (with a history of size $\tau$). Mutual I. is the Mutual Information associated to the $\delta\tau$ prediction. For all metrics, the higher the better. The best results are indicated in bold and the second best in italics.  | Acc. | Acc. $\delta\tau$ | Mutual I.
---|---|---|---
PCMCIG | 0.287 $\pm$0.000 | 0.471 $\pm$0.000 | 0.004 $\pm$0.000
PCMCI+GCNG | 0.588 $\pm$0.008 | 0.517 $\pm$0.005 | 0.143 $\pm$0.025
PCMCI+GATG | 0.482 $\pm$0.015 | 0.308 $\pm$0.045 | 0.111 $\pm$0.009
PCMCI+GATv2G | 0.567 $\pm$0.012 | 0.450 $\pm$0.017 | 0.116 $\pm$0.004
PCMCI${}_{\mathcal{G_{\overline{\text{corr}}}}}$ | 0.287 $\pm$0.000 | 0.471 $\pm$0.001 | 0.004 $\pm$0.000
PCMCI+GCN${}_{\mathcal{G_{\overline{\text{corr}}}}}$ | 0.582 $\pm$0.009 | 0.511 $\pm$0.006 | 0.126 $\pm$0.000
PCMCI+GAT${}_{\mathcal{G_{\overline{\text{corr}}}}}$ | 0.494 $\pm$0.019 | 0.300 $\pm$0.032 | 0.125 $\pm$0.008
PCMCI+GATv2${}_{\mathcal{G_{\overline{\text{corr}}}}}$ | 0.548 $\pm$0.014 | 0.434 $\pm$0.009 | 0.107 $\pm$0.015
LSTM | 0.565 $\pm$0.007 | 0.547 $\pm$0.011 | 0.190 $\pm$0.009
Transformer | 0.343 $\pm$0.068 | 0.332 $\pm$0.065 | 0.145 $\pm$0.011

###  4.4. Individual-Level Simulation

In this section, we aim to study the behaviour of an individual when simulated
for a longer period of time. Using our models, we generated new time-series
data of the same length as the available ground-truth time-series. The initial
state and context information are extracted from the ground truth. The models
then run partially autoregressively. Indeed, as in the previous section, we
focus on the behaviour of a single individual while the rest of the group is
unaffected. We generate a sequence $s_{t+1},s_{t+2},\dots$ while the context
$C_{t+1}^{*},C_{t+1}^{*},\dots$ remains unchanged.

#### Clustering

We sample short snippets (of $\tau+1$ timesteps) from each time-series and
concatenate the vectors of each timestep to build a single vector embedding of
the snippet. We project the resulting vector on a 2D space with
Multidimensional Scaling (MDS) (Borg and Groenen, 2005) and categorise the
data into $K=4$ clusters with K-Means. The objective is to observe whether the
generated short sequences of behaviours follow similar patterns to those in
the ground truth. The results are shown in Figure 5. The visualisation
obtained for the ground truth data are very different from that obtained with
the simulations. The vast majority of the true video snippets are well
separated from each other, and the samples in the cluster on the right of the
image are structured as concentric circles. In contrast, the simulated samples
do not form any organised structures. All models generate similar patterns.
The samples from all the models are either concentrated into a large convex
shape or into one or two smaller shapes. This result visually illustrates the
lack of information regarding the inner mechanisms of the individual agents,
as shown in the Mutual Information score of Table 2.

(a) Ground truth.

(b) PCMCI.

(c) PCMCI+GCN.

(d) PCMCI+GAT.

(e) PCMCI+GATv2.

(f) LSTM.

(g) Transformer.

(h) PCMCIC.

(i) PCMCI+GCNC.

(j) PCMCI+GATC.

(k) PCMCI+GATv2C.

(l) LSTMC.

(m) TransformerC.

Figure 5. Projection on a 2D space and K-Means clustering of the generated and
true data samples. The ground truth is shown on the top left corner. The
results of individual-level simulations are shown from Figures 5(b) to 5(g).
The results of group-level simulations are shown from Figures 5(h) to 5(m) and
indicated with a C.

#### Discrimination

To quantify the difference between the ground truth and the simulated
behaviours, we trained a discriminator model to classify true and counterfeit
data. Table 3 shows these results. We focus on the first half of the table. On
average, the LSTM discriminator reaches a higher accuracy than the Transformer
discriminator. The accuracy is also more consistent, with a systematically
smaller standard deviation by one order of magnitude. As for the Transformer
prediction model, data scarcity is a likely explanation for the lower
discrimination performance. We now focus on the performance of the generation
models to fool the discriminator. The GNN models achieve better performance
than the baseline prediction engines with the LSTM discriminator, with
PCMCI+GATv2G achieving the best score. The LSTM achieves the best performance
with the Transformer discriminator. The GNN models perform better on the most
challenging discriminator, indicating that they generated more plausible data
than the baseline models. Surprisingly, PCMCI achieved a better score than the
LSTM and Transformer. To generate plausible data, knowing the causal
mechanisms seems to be more important than accurately predicting the next
timestep.

Table 3. Accuracy of the discriminator. The mean and standard deviation across 3 training runs is shown. The discriminator is tasked to distinguish real samples from simulated samples. We evaluate the ability of the models to generate samples similar to the true distribution and fool the discriminator. Therefore, the lower accuracy the better. The best results are indicated in bold and the second best in italics.  | LSTM-Discr. | Transformer-Discr.
---|---|---
PCMCIG | 0.888 $\pm$0.004 | 0.892 $\pm$0.039
PCMCI+GCNG | 0.875 $\pm$0.001 | 0.823 $\pm$0.098
PCMCI+GATG | 0.880 $\pm$0.007 | 0.880 $\pm$0.018
PCMCI+GATv2G | 0.874 $\pm$0.003 | 0.893 $\pm$0.039
LSTM | 0.893 $\pm$0.005 | 0.786 $\pm$0.161
Transformer | 0.892 $\pm$0.004 | 0.818 $\pm$0.077
PCMCIGC | 0.873 $\pm$0.005 | 0.591 $\pm$0.043
PCMCI+GCNGC | 0.866 $\pm$0.002 | 0.543 $\pm$0.060
PCMCI+GATGC | 0.865 $\pm$0.002 | 0.660 $\pm$0.021
PCMCI+GATv2GC | 0.866 $\pm$0.001 | 0.839 $\pm$0.041
LSTMC | 0.888 $\pm$0.002 | 0.649 $\pm$0.150
TransformerC | 0.887 $\pm$0.001 | 0.732 $\pm$0.164

(a) Ground truth.

(b) PCMCIC.

(c) PCMCI+GCNC.

(d) LSTMC.

(e) TransformerC.

Figure 6. Sankey diagram of the behaviour flows. The flow diagrams are shown
for the series generated by a group of simulated meerkats.

###  4.5. Group-Level Simulation

In this section, we study the prediction abilities of our model at the group-
level. All individuals in the group are modelled such that the behaviour
$s_{t}^{(k)}$ of one individual $k$ will affect the context information
$C_{t+1}^{(j\neq k)}$ of the other individuals. We labelled the time-series
generated for an entire group with a C to differentiate them from the time-
series for an individual.

#### Discrimination

As for single individuals, we quantitatively evaluate the similarity between
the true and the simulated data. Table 3 shows the results. We focus on the
second half of the table. The prediction models achieved better performance at
the group-level when compared to single-individual time-series. This indicates
that the behaviours do not collapse under the influence of the other
predictions and maintain their initial proximity to the true behaviour
distribution. Moreover, the influence of the modified context improves the
quality of the generation. The interactions between individuals seem to be of
paramount importance for accurately modelling their behaviours. As in the
single-individual case, the LSTM discriminator distinguishes the true and
counterfeit data better than the Transformer discriminator, with a lower
standard deviation. PCMCI+GATGC and PCMCI+GCNGC achieve the best results with
the LSTM and Transformer discriminators, respectively. Overall, the GNN models
achieve similar or better performances than the baselines, with the exception
of PCMCI+GATv2GC with the Transformer discriminator. PCMCI even achieves the
second best score with this discriminator. This is consistent with the
observations made in the individual-level case, although causal models take
better advantage of group-level generation than the baseline models.

#### Visualisation

As for single individuals, we visualise the clusters generated for short
sequences of behaviours. The results are shown in Figure 5. There are no
apparent differences between the individual- and group-generated data. The
samples generated at the group-level do not form organised structures and
express patterns similar to their individual-level counterparts. In addition,
we visualise the Sankey diagrams of the generated series in Figure 6. A Sankey
diagram measures the flow between processes. In our case, it corresponds to
the distribution of the behaviour transitions, i.e. what are the most likely
behaviours one individual might take if in the current state. The ground truth
Sankey diagram greatly differs from the other diagrams. The majority of the
transitions are directed towards the same behaviour, and few transitions
actually lead to other behaviours. This is consistent with the observations
made in Section 4.1. In contrast, the simulated series are more volatile. One
behaviour often leads to new behaviours. LSTMC and TransformerC exhibit
similar patterns. In particular, the ”moving” and ”interacting with foreign
object” behaviours occur more often than in the ground truth data. One
potential explanation is overfitting to the most observed behaviours in the
training data to maximise the prediction accuracy. As shown in Figure 4,
”moving” and ”interacting with foreign object” are frequently observed
behaviours. Interestingly, the ”foraging” behaviour has little input and
output flows, despite being often observed in the ground truth. PCMCIC shows
patterns similar to those of the baselines, although the behaviours with the
highest flows differ. For instance, ”raised guarding” has a high input flow,
despite being scarcely observed in the data. This is not surprising, as the
model aims to maximise the strength of the causal dependencies and not the
likelihood of the behaviour transitions. However, there seems to be an
important overlap between these two objectives. Finally, PCMCI+GCNC shows a
very different pattern from the other diagrams. The behaviours are balanced
and the model does not provide higher flows to some behaviours. This can be a
possible explanation for the highest discrimination scores despite achieving a
prediction performance similar to that of the baselines.

###  4.6. Explainability

The PCMCI algorithm generates a causal graph that links cause variables to the
variables affected by them. Such causal graph allows the building of a sparse
agent model with greater interpretability than fully-connected models. It can
provide insight into what behaviours can lead to another, which outcomes are
not possible, and what context causes can influence one’s behaviour. Figure 7
shows an example of causal graph generated by PCMCI. Only the links towards
the state variables and with the highest causation strength are displayed. For
instance, a ”raised guarding” agent might start to ”playfight” in the
following timesteps; having neighbours ”moving” or ”interacting with a foreign
object” can lead an agent to follow them. Further in-depth investigation into
the structures of the causal graphs may lead to discoveries on the topic of
interest, but this is beyond the scope of this paper.

Figure 7. Causal graph generated by PCMCI. Only edges directed to state
variables and with the highest causation strength are shown. Context variables
with no directed edges with high strength to state variables are hidden.

##  5\. Conclusion

Understanding the complex behaviours of social animals from observations is an
important task encompassing numerous fields of study and applications, ranging
from the welfare of animals in zoos to the understanding of animal social
structures. This is a challenging task as the data collection process is
expensive and may not cover the full scope of the causes affecting an
individual’s behaviour, in particular, the mental states of an individual are
unknown.

In our work, we tackle the problem of modelling the behaviours of a group of
meerkats interacting together in a zoo environment. We use causality theory
and graph neural networks to build an interpretable prediction and generation
engine. We show that our method can compete with and outperform standard deep
learning models with a higher number of parameters. In particular, the
proposed method can generate more realistic simulation data than the
baselines. This work is a first step in the modelling of social animals, and
many challenges are yet to be explored. This paper uncovers some limitations
of the proposed and current models, such as the lack of information (in
Shannon’s definition) learned by the models and the large difference between
the ground truth and simulated samples. We highlight a discrepancy between
accuracy performance at the statistical level, and accurate modelling of the
inner mechanisms of the agents.

In our future work, we will focus on learning new behaviours instead of
relying on a pre-defined set. We will combine our current method with
dynamical models to learn the movements of the agents in addition to their
behaviours and study how it affects the level of information learned by the
models. Finally, we will use Inverse Reinforcement Learning to model the long-
term policies and rewards that drive the agent’s behaviours.

## References

  * (1)
  * Borg and Groenen (2005) Ingwer Borg and Patrick JF Groenen. 2005.  _Modern multidimensional scaling: Theory and applications_.  Springer Science & Business Media.
  * Brody et al. (2022) Shaked Brody, Uri Alon, and Eran Yahav. 2022.  How Attentive are Graph Attention Networks?. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.  https://openreview.net/forum?id=F72ximsx7C1
  * Buskirk et al. (1981) Ruth E Buskirk, Cliff Frohlich, and Gary V Latham. 1981.  Unusual animal behavior before earthquakes: A review of possible sensory mechanisms.  _Reviews of geophysics_ 19, 2 (1981), 247–270.
  * Chen et al. (2020) Duxin Chen, Mingyu Kang, and Wenwu Yu. 2020.  Probabilistic causal inference for coordinated movement of pigeon flocks.  _Europhysics Letters_ 130, 2 (2020), 28004.
  * Clutton-Brock et al. (2004) TH Clutton-Brock, AF Russell, and LL Sharpe. 2004.  Behavioural tactics of breeders in cooperative meerkats.  _Animal Behaviour_ 68, 5 (2004), 1029–1040.
  * Colombo et al. (2012) Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. 2012.  Learning high-dimensional directed acyclic graphs with latent and selection variables.  _The Annals of Statistics_ (2012), 294–321.
  * Dougherty et al. (2018) Eric R Dougherty, Dana P Seidel, Colin J Carlson, Orr Spiegel, and Wayne M Getz. 2018.  Going through the motions: incorporating movement analyses into disease research.  _Ecology letters_ 21, 4 (2018), 588–604.
  * Entner and Hoyer (2010) Doris Entner and Patrik O Hoyer. 2010.  On causal discovery from time series data using FCI.  _Probabilistic graphical models_ (2010), 121–128.
  * Gendron et al. (2023) Gaël Gendron, Michael Witbrock, and Gillian Dobbie. 2023.  A Survey of Methods, Challenges and Perspectives in Causality.  _CoRR_ abs/2302.00293 (2023).  https://doi.org/10.48550/arXiv.2302.00293 arXiv:2302.00293
  * Hinde (1976) R. A. Hinde. 1976.  Interactions, Relationships and Social Structure.  _Man_ 11, 1 (1976), 1–17.  http://www.jstor.org/stable/2800384
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ 9, 8 (1997), 1735–1780.  https://doi.org/10.1162/neco.1997.9.8.1735
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.  https://openreview.net/forum?id=SJU4ayYgl
  * Kreer (1957) J. Kreer. 1957.  A question of terminology.  _IRE Transactions on Information Theory_ 3, 3 (1957), 208–208.  https://doi.org/10.1109/TIT.1957.1057418
  * Loh (2011) Wei-Yin Loh. 2011.  Classification and regression trees.  _Wiley interdisciplinary reviews: data mining and knowledge discovery_ 1, 1 (2011), 14–23.
  * Lord et al. (2016) Warren M. Lord, Jie Sun, Nicholas T. Ouellette, and Erik M. Bollt. 2016.  Inference of Causal Information Flow in Collective Animal Behavior.  _IEEE Trans. Mol. Biol. Multi Scale Commun._ 2, 1 (2016), 107–116.  https://doi.org/10.1109/TMBMC.2016.2632099
  * Moraffah et al. (2021) Raha Moraffah, Paras Sheth, Mansooreh Karami, Anchit Bhattacharya, Qianru Wang, Anique Tahir, Adrienne Raglin, and Huan Liu. 2021.  Causal inference for time series analysis: problems, methods and evaluation.  _Knowl. Inf. Syst._ 63, 12 (2021), 3041–3085.  https://doi.org/10.1007/s10115-021-01621-0
  * Morales et al. (2010) Juan M Morales, Paul R Moorcroft, Jason Matthiopoulos, Jacqueline L Frair, John G Kie, Roger A Powell, Evelyn H Merrill, and Daniel T Haydon. 2010.  Building the bridge between animal movement and population dynamics.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 365, 1550 (2010), 2289–2301.
  * Pearl (2009a) Judea Pearl. 2009a.  _Causality_ (2 ed.).  Cambridge University Press.  https://doi.org/10.1017/CBO9780511803161
  * Pearl (2009b) Judea Pearl. 2009b.  _Causality_.  Cambridge university press.
  * Pearl (2014) Judea Pearl. 2014.  The Deductive Approach to Causal Inference.  _Journal of Causal Inference_ 2, 2 (2014), 115–129.  https://doi.org/doi:10.1515/jci-2014-0016
  * Porfiri (2018) Maurizio Porfiri. 2018.  Inferring causal relationships in zebrafish-robot interactions through transfer entropy: a small lure to catch a big fish.  _Animal Behavior and Cognition_ 5, 4 (2018), 341–367.
  * Rew et al. (2019) Jehyeok Rew, Sungwoo Park, Yongjang Cho, Seungwon Jung, and Eenjun Hwang. 2019.  Animal movement prediction based on predictive recurrent neural network.  _Sensors_ 19, 20 (2019), 4411.
  * Rogers et al. (2023) Mitchell Rogers, Gaël Gendron, David Arturo Soriano Valdez, Mihailo Azhar, Yang Chen, Shahrokh Heidari, Caleb Perelini, Padriac O’Leary, Kobe Knowles, Izak Tait, Simon Eyre, Michael Witbrock, and Patrice Delmas. 2023.  Meerkat Behaviour Recognition Dataset.  _CoRR_ abs/2306.11326 (2023).  https://doi.org/10.48550/arXiv.2306.11326 arXiv:2306.11326
  * Rose and Riley (2021) Paul E. Rose and Lisa M. Riley. 2021.  Conducting Behavioural Research in the Zoo: A Guide to Ten Important Methods, Concepts and Theories.  _Journal of Zoological and Botanical Gardens_ 2, 3 (2021), 421–444.  https://doi.org/10.3390/jzbg2030031
  * Rubin (1974) Donald B Rubin. 1974.  Estimating causal effects of treatments in randomized and nonrandomized studies.  _Journal of educational Psychology_ 66, 5 (1974), 688.
  * Runge et al. (2019a) Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. 2019a.  Detecting and quantifying causal associations in large nonlinear time series datasets.  _Science advances_ 5, 11 (2019), eaau4996.
  * Runge et al. (2019b) Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. 2019b.  Detecting and quantifying causal associations in large nonlinear time series datasets.  _Science Advances_ 5, 11 (2019), eaau4996.  https://doi.org/10.1126/sciadv.aau4996 arXiv:https://www.science.org/doi/pdf/10.1126/sciadv.aau4996
  * Scarselli et al. (2009) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009.  The Graph Neural Network Model.  _IEEE Trans. Neural Networks_ 20, 1 (2009), 61–80.  https://doi.org/10.1109/TNN.2008.2005605
  * Schölkopf et al. (2021) Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021.  Toward Causal Representation Learning.  _Proc. IEEE_ 109, 5 (2021), 612–634.  https://doi.org/10.1109/JPROC.2021.3058954
  * Scott (2014) Katy Scott. 2014.  _Behaviour and Endocrinology of Meerkats in Zoos_.  Ph.D. Dissertation. University of Exeter.  http://hdl.handle.net/10871/16393
  * Scott et al. (2017) Katy Scott, Michael Heistermann, Michael A Cant, and Emma I K Vitikainen. 2017.  Group size and visitor numbers predict faecal glucocorticoid concentrations in zoo meerkats.  _R Soc Open Sci_ 4, 4 (April 2017), 161017.
  * Shamoun-Baranes et al. (2012) Judy Shamoun-Baranes, Roeland Bom, E Emiel van Loon, Bruno J Ens, Kees Oosterbeek, and Willem Bouten. 2012.  From sensor data to animal behaviour: an oystercatcher example.  _PloS one_ 7, 5 (2012), e37997.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ 27, 3 (1948), 379–423.
  * Shettleworth (2001) Sara J. Shettleworth. 2001.  Animal cognition and animal behaviour.  _Animal Behaviour_ 61, 2 (2001), 277–286.  https://doi.org/10.1006/anbe.2000.1606
  * Shettleworth (2009) Sara J Shettleworth. 2009.  _Cognition, evolution, and behavior_.  Oxford university press.
  * Spirtes and Glymour (1991) Peter Spirtes and Clark Glymour. 1991.  An Algorithm for Fast Recovery of Sparse Causal Graphs.  _Social Science Computer Review_ 9, 1 (1991), 62–72.  https://doi.org/10.1177/089443939100900106 arXiv:https://doi.org/10.1177/089443939100900106
  * Spirtes et al. (2000) Peter Spirtes, Clark N Glymour, and Richard Scheines. 2000.  _Causation, prediction, and search_.  MIT press.
  * Splawa-Neyman et al. (1990) Jerzy Splawa-Neyman, Dorota M Dabrowska, and Terrence P Speed. 1990.  On the application of probability theory to agricultural experiments. Essay on principles. Section 9.  _Statist. Sci._ (1990), 465–472.
  * Sumpter et al. (2012) David JT Sumpter, Richard P Mann, and Andrea Perna. 2012.  The modelling cycle for collective animal behaviour.  _Interface focus_ 2, 6 (2012), 764–773.
  * Timberlake (1997) William Timberlake. 1997.  An animal-centered, causal-system approach to the understanding and control of behavior.  _Applied Animal Behaviour Science_ 53, 1-2 (1997), 107–129.
  * Valletta et al. (2017) John Joseph Valletta, Colin Torney, Michael Kings, Alex Thornton, and Joah Madden. 2017.  Applications of machine learning in animal behaviour studies.  _Animal Behaviour_ 124 (2017), 203–220.
  * Van Moorter et al. (2016) Bram Van Moorter, Christer M Rolandsen, Mathieu Basille, and Jean-Michel Gaillard. 2016.  Movement is the glue connecting home ranges and habitat selection.  _Journal of Animal Ecology_ 85, 1 (2016), 21–31.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 5998–6008.  https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  * Velickovic et al. (2018) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks. In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.  https://openreview.net/forum?id=rJXMpikCZ
  * Wakefield (2013) Monica L Wakefield. 2013.  Social dynamics among females and their influence on social structure in an East African chimpanzee community.  _Animal Behaviour_ 85, 6 (2013), 1303–1313.
  * Xia et al. (2021) Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. 2021.  The Causal-Neural Connection: Expressiveness, Learnability, and Inference. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 10823–10836.  https://proceedings.neurips.cc/paper/2021/hash/5989add1703e4b0480f75e2390739f34-Abstract.html
  * Yao et al. (2021) Liuyi Yao, Zhixuan Chu, Sheng Li, Yaliang Li, Jing Gao, and Aidong Zhang. 2021.  A Survey on Causal Inference.  _ACM Trans. Knowl. Discov. Data_ 15, 5, Article 74 (may 2021), 46 pages.  https://doi.org/10.1145/3444944
  * Zecevic et al. (2021) Matej Zecevic, Devendra Singh Dhami, Petar Velickovic, and Kristian Kersting. 2021.  Relating Graph Neural Networks to Structural Causal Models.  _CoRR_ abs/2109.04173 (2021).  arXiv:2109.04173 https://arxiv.org/abs/2109.04173
  * Zhang et al. (2022) Tao Zhang, Haoran Shan, and Max A. Little. 2022.  Causal GraphSAGE: A robust graph method for classification based on causal sampling.  _Pattern Recognit._ 128 (2022), 108696.  https://doi.org/10.1016/j.patcog.2022.108696


Luca Crupi, Elia Cereda, and Daniele Palossi 11institutetext: Dalle Molle
Institute for Artificial Intelligence, USI-SUPSI, Lugano, Switzerland,
22institutetext: Integrate Systems Laboratory, ETH Zürich, Zürich, Switzerland
22email<EMAIL_ADDRESS>
# Fusing Multi-sensor Input with State Information on TinyML Brains for
Autonomous Nano-drones

Luca Crupi 11 Elia Cereda 11 Daniele Palossi 1212

###### Abstract

Autonomous nano-drones ($\sim$$10\text{\,}\mathrm{cm}$ in diameter), thanks to
their ultra-low power TinyML-based brains, are capable of coping with real-
world environments. However, due to their simplified sensors and compute
units, they are still far from the sense-and-act capabilities shown in their
bigger counterparts. This system paper presents a novel deep learning-based
pipeline that fuses multi-sensorial input (i.e., low-resolution images and
8$\times$8 depth map) with the robot’s state information to tackle a human
pose estimation task. Thanks to our design, the proposed system – trained in
simulation and tested on a real-world dataset – improves a state-unaware
State-of-the-Art baseline by increasing the $R^{2}$ regression metric up to
0.10 on the distance’s prediction.

###### keywords:

Sensor fusion, TinyML, Autonomous nano-drones

##  1 Introduction

Miniaturized autonomous nano-drones, from sub-$30\text{\,}\mathrm{g}$ blimps
[6] to palm-sized quadrotors [2], are becoming increasingly popular in
academia and industry thanks to their broad applicability [8]. They can assist
human operators in rescue missions, inspect narrow places, and monitor indoor
facilities. Additionally, thanks to their small footprint and weight, they can
safely operate close to people, being harmless. However, the vast versatility
of autonomous nano-drones comes at the price of simple and cheap electronics,
leading to ultra-constrained computational resources and simplistic sensors,
e.g., low-resolution cameras.

Recent works [2, 4, 9] have shown how, despite their limitations, nano-drones
can achieve a high level of intelligence using tiny deep learning (DL) models
fed with multiple sensors and relying only on sub-$100\text{\,}\mathrm{mW}$
System-on-Chips (SoCs). The vast majority of these works address DL-based
perception tasks focusing on understanding the surrounding environment, such
as estimating the relative pose of an object or a human [2, 4] or avoiding
obstacles [5, 9]. We define these as allocentric tasks, in which the
prediction refers to an external object/subject, in contrast to egocentric
tasks, in which the subject is the robot itself.

In DL models for egocentric tasks, such as state estimation and visual
odometry [1, 3, 7], exploiting the robot’s state as an auxiliary input is
common. However, whether it can also benefit allocentric tasks is still an
open research question, which we aim to explore by introducing state
information into a convolutional neural network (CNN) for the allocentric task
of human pose estimation. We start from a State-of-the-Art (SoA) baseline CNN,
running on a Greenwaves Technologies (GWT) GAP8 SoC and fed with 160$\times$96
pixel images and an 8$\times$8 depth map, both acquired aboard a nano-drone
and used to estimate the distance ($x$) and the lateral displacement ($y$)
between the drone and a human in front of it. Then, we extend this model by
feeding part of the nano-drone’s state as an additional input of the CNN, and
we present an ablation study exploring i) how different state information
(i.e., only pitch or roll, and pitch+roll) and ii) how different fusion
techniques affect the final CNN’s regression performance.

Our exploration leads to four alternative fusion approaches, resulting in just
as many CNNs we deploy and characterize on the GAP8 SoC aboard our target
nano-drone. Despite the CNNs training being done only in simulation, our
results from a real-world test set (more than $3.5\text{\,}\mathrm{k}$ image-
depth pairs) highlight how the introduction of either of the state angles
improves the regression performance of our neural network, measured with an
$R^{2}$ increase of +0.10 on $x$, and +0.02 on $y$, at an increase of the
computational cost, measured with the number of multiply-and-accumulate (MAC)
operations, of just 0.11% if compared with the state-unaware SoA baseline
model [2].

##  2 System description

Our work starts from the state-unaware vision+depth CNN presented in [2] for
the human pose estimation task. We share the same target nano-drone, i.e., the
Bitcraze Crazyflie 2.1, and the same computational and sensorial resources,
such as the STM32 microcontroller, the GWT GAP8 octa-core SoC, the Himax
HM01B0 monocular grayscale QVGA camera, and the ST VL53LC5CX Time-of-Flight
(ToF) multi-zone ranging sensor. Like in [2], we process grayscale $160\times
96$ images and 8$\times$$8\text{\,}\mathrm{p}\mathrm{x}$ depth maps, while we
predict the $(x,y)$ position coordinates of the human subject, as shown in
Fig. 1. The depth information is fused with a mid fusion approach (i.e., the
best-performing in [2]) that introduces the depth map as one of the feature
maps in input to the CNN’s 6th convolutional layer.

Our state-aware CNNs use the roll ($\varphi$) and pitch ($\theta$) components
of the drone attitude, with the two alternative representations shown in Fig.
1-A: either a two-dimensional state map in which the elements repeat in a
Bayer-like pattern, or a two-element ($\varphi,\theta$) state vector. The
former representation is more suitable for fusion approaches in which the
state should be spread across multiple neural connections, such as in Fig.
1-B1 called input fusion, and in Fig. 1-B2 called mid fusion. The sizes of the
two-state maps vary to match each convolutional layer’s input dimensions,
i.e., $160\times 96$ and $10\times 6$. Employing a two-element state vector,
instead, is a cheaper solution – i.e., less memory and MAC operations –
suitable for introducing the state directly into the last CNN’s fully
connected layer, depicted in Fig. 1-B3 and called late fusion (direct), or
after processing it with a 2-layer multilayer perceptron (MLP), Fig. 1-B4. As
in [2], for all variants, input dropout is employed at training time to use
all three inputs equally, i.e., camera, depth, and state.

Finally, we train and validate the four models with the same
$700\text{\,}\mathrm{k}$-sample dataset (from the Webots simulator) and
procedures as in [2]. We apply domain randomization, varying the environment
(walls, floors, and decoy objects) and the human subject (chosen among 27 3D
models). Our procedure randomly selects the drone’s attitude for each sample,
such that our state variables (roll and pitch) are uniformly distributed over
the $[$$,$$]$ range. All CNNs are trained with stochastic gradient descent at
a learning rate of 0.001 for 100 epochs, selecting the model with the lowest
validation loss. We evaluate the models’ performance on a challenging
3500-sample real-world test set, collected in a room equipped with a motion
capture system, where the drone flights stress roll and pitch in the same
$[$$,$$]$ range.

Figure 1: A) Our CNN inputs. B) CNN architecture exploration, based on the SoA
vision+depth backbone from [2]. Our proposed state-aware models either with 1)
input fusion, 2) mid fusion, 3) late fusion (direct), and 4) late fusion with
MLP.

##  3 Experimental results

In this section, we present our ablation study on the proposed fusion
techniques, i.e., input, mid, and two late fusions (MLP and direct), combining
them with different states (i.e., pitch, roll, pitch+roll) and the effect of a
dropout layer applied on the state during training. Each combination results
in a different model (24 in total) compared against the state-unaware SoA
baseline proposed in [2]. Each model is trained 5 times, resulting in 125
distinctive instances, presented in Fig. 2 and evaluated with the $R^{2}$
score on the $x$ and $y$ output variables. The $R^{2}$ score measures
regression performance in a normalized range [$-\infty$, 1]. A perfect
predictor scores 1, while a dummy predictor that always predicts the average
of the test set would score 0.

Figure 2: $R^{2}$ comparison of our fusion techniques (i.e., input, mid,
direct, and MLP) with and without dropout and with different states (i.e.,
pitch, roll, and pitch+roll) vs. the state-unaware SoA baseline [2]. Each
marker is the average of 5 different training.

The results in Fig. 2 highlight that introducing any state is consistently
beneficial (with the only exception of input fusion without dropout and
pitch+roll states), regardless of dropout and fusion technique, with the sum
of the $R^{2}$ improvements on $x$ and $y$ up to +0.11 w.r.t. the state-
unaware SoA baseline model. Applying the dropout on the state input does not
affect the $R^{2}$ standard deviation, contrary to the results from [2]. In
this regard, input dropout in [2] was used on the vision and depth sensorial
inputs to force the network to consider both information streams. Instead, in
this case, the model does not overfit on the state even without dropout
because it would be insufficient to estimate the person’s position without a
camera or ToF sensor. Out of all CNNs, the best-performing model uses only the
pitch state, without dropout and with late fusion (direct). Its $R^{2}$
increases by +0.10 on the $x$ and by +0.01 on the $y$ output w.r.t. the state-
unaware SoA baseline, with a negligible increase in the computational cost,
i.e., 4 MAC. The configuration input fusion on pitch+roll without dropout is
the only one where the performance slightly decreases on $x$ (-0.01 $R^{2}$).

| SoA baseline | Input | Mid | Direct | MLP
---|---|---|---|---|---
$R^{2}$ (peak $x$/$y$) | 0.35/0.46 | 0.45/0.45 | 0.43/0.47 | 0.45/0.48 | 0.44/0.48
Memory [$\text{\,}\mathrm{kB}$] | 304.4 | 305.2 | 305.5 | 304.4 | 304.8
MAC/frame [$\text{\,}\mathrm{M}$] | 14.7 | 17.8 | 14.8 | 14.7 | 14.7

Table 1: $R^{2}$ peak performance, memory occupancy, and computational cost
for the state-unaware SoA baseline and the four different state fusion
techniques. We report the maximum $R2$ for each variable, $x$ and $y$, and for
each fusion method.

Tab. 1 further reports the $R^{2}$ peak performance for each output ($x$ and
$y$ from different models), the memory occupancy, and the computational cost
for each state fusion technique and for the state-unaware SoA baseline. Among
the four architecture variants, input fusion and direct late fusion achieve
the highest regression performance improvements (+0.10 and +0.11,
respectively) w.r.t the SoA baseline. However, input fusion is the most
expensive in computational and second most expensive in memory cost,
respectively
+$3.1\text{\,}\mathrm{M}\mathrm{M}\mathrm{A}\mathrm{C}\mathrm{/}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}$
(+21%) and +$0.8\text{\,}\mathrm{kB}$ (+2.6%) w.r.t. the baseline. The other
three variants (i.e., mid, MLP, and direct fusion) increase the MAC workload
by less than 0.11%. However, the two late-fusion variants require less memory
compared to mid-fusion (+1.3% increase vs. +3.6%, respectively). Thus,
considering the trade-off between regression performance and memory and
computational costs suggests that the direct late-fusion model is the most
convenient. These results highlight the benefit provided by the fusion of the
state in the pose estimation task with minimal or no computational cost
addition (between 0 and 0.11% more MAC for the mid, direct, and MLP
approaches) and with a limited increase of the memory usage, i.e., between 0
and 0.36% w.r.t. the SoA baseline.

##  4 Conclusion

This work introduces a novel study on how to fuse state information into a
lightweight multi-sensorial (camera and depth map) CNN for the allocentric
task of human pose estimation aboard an autonomous nano-drone. Our analysis
highlights the impact on the regression performance of different states (roll,
pitch, roll+pitch) and fusion techniques (input, mid-fusion, and late
fusions). Our key findings consistently show the benefit of the state fusion:
mean $R^{2}$ improvement of 0.06 on the $x$ variable w.r.t. the SoA baseline
model. Our best model, the late fusion approach, increases the $R^{2}$ up to
0.10 and 0.01 on $x$ and $y$ with a negligible overhead in memory or
computation.

## References

  * [1] R. Clark et al.  VINet: Visual-inertial odometry as a sequence-to-sequence learning problem.  Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017.
  * [2] L. Crupi et al.  Sim-to-real vision-depth fusion CNNs for robust pose estimation aboard autonomous nano-quadcopters.  In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7711–7717, 2023.
  * [3] L. Han et al.  DeepVIO: Self-supervised deep learning of monocular visual inertial odometry using 3d geometric constraints.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6906–6913, 2019.
  * [4] E. Kaufmann et al.  Beauty and the beast: Optimal methods meet learning for drone racing.  In 2019 International Conference on Robotics and Automation (ICRA), pages 690–696. IEEE, 2019.
  * [5] C. Liu et al.  Nano quadcopter obstacle avoidance with a lightweight monocular depth network.  IFAC-PapersOnLine, 56(2):9312–9317, 2023.  22nd IFAC World Congress.
  * [6] D. Palossi et al.  Self-sustainability in nano unmanned aerial vehicles: A blimp case study.  In Proceedings of the computing frontiers conference, pages 79–88, 2017.
  * [7] D. Palossi et al.  Ultra low-power visual odometry for nano-scale unmanned aerial vehicles.  In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017, pages 1647–1650, 2017.
  * [8] H. Shakhatreh et al.  Unmanned Aerial Vehicles (UAVs): A Survey on Civil Applications and Key Research Challenges.  IEEE access : practical innovations, open solutions, 7:48572–48634, 2019.
  * [9] N. Zhang et al.  End-to-end nano-drone obstacle avoidance for indoor exploration.  Drones, 8(2):33, 2024.