[1,1,2,3,4,5,6,7,8,9,1]Daniele Marinazzo,Jan Van Roozendaal,Fernando E. Rosas,
Massimo Stella,Renzo Comolatti, Nigel Colenbier, Sebastiano Stramaglia, Yves
Rosseel Department of Data Analysis, Ghent University, Belgium,Data Science
Institute, Imperial College London, London, UK, Centre for Psychedelic
Research, Imperial College London, London, UK, Centre for Complexity Science,
Imperial College London, London, UK, Department of Informatics, University of
Sussex, Brighton, UK, CogNosco Lab, Dept. of Computer Science, University of
Exeter, UK, Department of Biomedical and Clinical Sciences “L. Sacco”,
Università degli Studi di Milano, Italy, IRCCS San Camillo Hospital, Via
Alberoni 70, Venice 30126, Italy, Physics Department, Universitá degli Studi
di Bari Aldo Moro, and INFN Sezione di Bari, Italy Marinazzo

# An information-theoretic approach to hypergraph psychometrics

###### Abstract

Psychological network approaches propose to see symptoms or questionnaire
items as interconnected nodes, with links between them reflecting pairwise
statistical dependencies evaluated on cross-sectional, time-series, or panel
data. These networks constitute an established methodology to assess the
interactions and relative importance of nodes/indicators, providing an
important complement to other approaches such as factor analysis. However,
focusing the modelling solely on pairwise relationships can neglect
potentially critical information shared by groups of three or more variables
in the form of higher-order interdependencies. To overcome this important
limitation, here we propose an information-theoretic framework based on
hypergraphs as psychometric models. As edges in hypergraphs are capable of
encompassing several nodes together, this extension can thus provide a richer
representation of the interactions that may exist among sets of psychological
variables. Our results show how psychometric hypergraphs can highlight
meaningful redundant and synergistic interactions on either simulated or
state-of-art, re-analyzed psychometric datasets. Overall, our framework
extends current network approaches while leading to new ways of assessing the
data that differ at their core from other methods, extending the psychometric
toolbox and opening promising avenues for future investigation.

###### keywords:

Psychological networks, psychometrics, hypergraphs, higher order interactions,
latent variables, information theory, synergy, redundancy

Daniele Marinazzo0000-0002-9803-0122
Fernando E. Rosas0000-0001-7790-6183
Massimo Stella0000-0003-1810-9699
Renzo Comolatti0000-0003-1097-3095
Nigel Colenbier0000-0003-0928-2668
Sebastiano Stramaglia0000-0002-5873-8564
Yves Rosseel0000-0002-4129-4477
Correspondence concerning this article should be addressed to Daniele
Marinazzo, Department of Data Analysis, Faculty of Psychological and
Educational Sciences, Ghent University, 1 Henri Dunantlaan, B-9000 Ghent,
Belgium. E-mail<EMAIL_ADDRESS>
##  1 Introduction

The last decade has seen the growth of various network approaches that allow
researchers to probe and exploit potential interactions of observable
variables associated with a given construct, or between constructs themselves
[57, 9, 68, 38, 25, 10]. These approaches build networks where nodes represent
measurable variables and links reflect relationships between them, with the
purpose of focusing not on common underlying factors but on the structure of
the inter-dependencies between the observables. Recently some approaches have
been developed to include also factors in the network [29], see below for
further details. Such psychometric networks are capable of reflecting
dynamical processes [57], and can consider lagged relationships that can
reveal dynamics leading to the emergence of symptoms [9]. Crucially, these
networks often reveal non-trivial features of the groups of variables they
represent, which can be identified by leveraging tools of network science such
as measures of centrality [12, 19] or community detection [68]. Furthermore,
it has been argued that important outgoing effects or events can be identified
from an examination of either the overall network or part of it — e.g. how
interactions between certain symptoms may create a vicious circle triggering a
series of negative events along the way [57].

An illustrative case of the potential of network approaches can be found in
the work on comorbidity [23], i.e. constructs that share multiple observables
(e.g. depression and anxiety, which are sometimes considered as different
facets of neuroticism [36, 68]), and can be studied in terms of common edges
or nodes, without limitations coming from modelling these sets via multiple
latent variables [16, 34].

While network properties such as centrality and modularity provide valuable
information about the overall topology of a network [10], traditional network
approaches are limited in a fundamental yet often unacknowledged way: as their
construction is based solely on pairwise links, they cannot give account of
potential higher-order (i.e. beyond pairwise) interactions that involve three
or more observables [40, 52, 32]. In effect, the interactions between three or
more variables are often nontrivial, poorly understood and, yet, can play
fundamental roles in driving complex systems [5]. From a statistical
perspective, a pairwise approach assessing the relation between two variables
(conditioned on other variables or not) cannot inform on the existence of
synergistic or redundant interactions, which by definition involve three or
more variables [40, 74]. Recent work is revealing how high-order
interdependencies play key roles in diverse hallmark aspects of complex
systems, including self-organisation [48], synchronisation [59], percolation
[30], and emergent phenomena in general [47].

Hypergraphs are the natural extension of networks that can properly give
account of high-order interdependencies. In effect, analogously as graphs,
hypergraphs are composed of nodes and hyperedges (or hyperlinks) connecting
them, with the key difference that hyperedges can connect more than two nodes
nodes [6]. Hyperedges can represent e.g. co-authors [43], components of an
ecosystem [3], or individuals sharing a social or physical space in contagion
models [60]. As an example, Figure 1 depicts an hypergraph and its binarized
counterpart.

Figure 1: Illustrative example of an hypergraph with 8 nodes and 4 hyperedges
(left) and its binarized version (right), generated with the R package HyperG
[37].

In the present work we put forward a psychometric framework that harnesses the
power of hypergraphs to represent high-order interdependencies. At its core,
our proposal is based on idea of moving from pairwise to a higher-order
interdependencies as described in the multivariate joint distribution of the
variables in question. As such, our proposed framework can be conceived as an
extension of existent psychometric network approaches that, in turn, enables a
systematic investigation of the role of redundant and synergistic interactions
in psychometric scenarios to better characterise complex emergent behavior
[50]. In doing this, our approach is not necessarily overlapping, but rather
complementary to the discovery of underlying structure such as latent factors
and interactions, even in a higher-order way as described in [29]. Our
proposed framework is based on the O-information [49], an information-
theoretic metric capable of individuating the interdependencies within
multiplets of three and more variables (i.e. hyperlinks of statistical
dependencies in the graph theory terminology), determining whether this common
information is predominantly redundant or synergistic based on the sign of the
informational content. The proposed framework is illustrated on synthetic
datasets where the ground truth is known, and on standard datasets currently
analyzed in network psychometrics studies.

##  2 Methods

###  2.1 The state of the art: Exploratory Data Analysis and Networks

Let us consider $n$ measurable variables $X_{1},\ldots,X_{n}$, which could
correspond to symptoms of a condition of interest, or items in a
questionnaire. An attractive way to describe the interdependencies between
these variables is to represent them as a network [69]. In general, networks
are defined by two elements: (a) a set of nodes/vertices denoted by $V$, and
(b) a collection of links/edges between the nodes denoted by $E$. Individual
edges are usually denoted as $\textit{e}_{ij}$, where $i$ and $j$ represent
the indices of two nodes. A network of observables typically represents each
variable as a node, and uses links to depict relationships between the
variables.

Among the simplest and most used ways for establishing a relationship between
two random variables ones finds (a) the Pearson correlation [40], and (b) the
partial Pearson correlation, conditioned against all other considered
variables [68]. However, both these approaches are affected by important
limitations. In effect, it has been convincingly shown that using plain
correlations to build network links has serious drawbacks in terms of
potentially highlighting spurious correlations [20, 8]. While this issue can
be partially avoided by adopting conditional correlations, these in turn can
also be problematic in the presence of colliders [46] and in general of shared
informational content [62]. Specifically, network analyses typically consider
networks whose links are given by the precision matrix $C=\textstyle\sum^{-1}$
with $\textstyle\sum$ being the correlation matrix, which allows to calculate
partial correlations via
$\rho_{ij}=\frac{C_{ij}}{\sqrt{(}C_{ii}\dot{C}_{jj})}$. It is worth mentioning
that accurate estimations of the precision matrix are non-trivial and prone to
biases, but adequate techniques have been developed in recent years [19]. In
particular, it is common to use regularization techniques such as LASSO [25,
24] to restrict the number of non-zero entries in the precision matrix.

###  2.2 Measuring higher-order effects via the $O$-information

While Shannon’s mutual information [58] may provide an encompassing account on
the interdependencies between two (sets of) variables, it is unable to fully
explore the rich interplay that can take place within triple or higher-order
interactions [40, 52, 32]. Two popular multivariate extensions of the Shannon
mutual information are the Total Correlation (TC) [72] and the Dual Total
Correlation (DTC) [64], which can be calculated as

$\displaystyle\text{TC}(X_{1},\dots,X_{n})$
$\displaystyle:=I(X_{1};X_{2})+I(\bm X_{1}^{2};X_{3})+\dots+I(\bm
X_{1}^{n-1};X_{n}),$ (1) $\displaystyle\text{DTC}(X_{1},\dots,X_{n})$
$\displaystyle:=I(X_{1};X_{2}|\bm X_{3}^{n})+I(\bm X_{1}^{2};X_{3}|\bm
X_{4}^{n})+\dots+I(\bm X_{1}^{n-1};X_{n})$ (2)

Above, $I(X;Y)$ corresponds to the Shannon mutual information between $X$ and
$Y$, $I(X;Y|Z)$ to the conditional mutual information given $Z$, and $\bm
X_{i}^{j}=(X_{i},\dots,X_{j})$ is a shorthand notation. Both the TC and DTC
are non-negative quantities that are zero if and only if all variables
$X_{1},\dots,X_{N}$ are jointly statistically independent — i.e. if
$p(X_{1},\dots,X_{n})=\prod_{i=1}^{n}p(X_{i})$. Interestingly, both are equal
to the mutual information for $n=2$ but differ when $n\geq 3$. In fact, both
metrics provide distinct but complementary views on multivariate
interdependencies: it can be shown that TC accounts for the effect of
_collective constraints_ (i.e. regions of the phase space that the system
explores less in its dynamics), while the DTC measures the amount of _shared
randomness_ between the variables (i.e. the information that can be collected
in one variable that also refers to the activity of another) [49].

An attractive way to exploit these complementary views is by considering their
difference,

$\displaystyle\Omega_{n}(X_{1},\dots,X_{n})$
$\displaystyle=\text{TC}(X_{1},\dots,X_{n})-\text{DTC}(X_{1},\dots,X_{n})\leavevmode\nobreak\
.$ (3)

which is known as the _O-information_ [49]. The O-information is a signed
metric that captures the balance between high- and low-order statistical
constraints: Low-order constraints impose strong restrictions on the system
and allow little amount of shared information between variables, while high-
order constraints impose collective restrictions that enable large amounts of
shared randomness. By construction, $\Omega(X_{1},\dots,X_{n})<0$ implies a
predominance of high-order constraints within the system of interest, a
condition that is usually referred to as statistical synergy. Conversely,
$\Omega(X_{1},\dots,X_{n})>0$ implies that the system is dominated by low-
order constraints, which imply redundancy of information. For more details
related to the O-information and its interpretation, we refer the reader to
[49].

###  2.3 Measuring higher-order effects on data

Let us consider a scenario where we have data from $n$ variables of interest,
denoted by $X_{1},\dots,X_{n}$. It is important to note that the O-information
of the whole, $\Omega(X_{1},\dots,X_{n})$, only provides information about the
dominant trend, and hence it is also interesting to calculate the
O-information of multiplets, i.e. subsets of variables, as they can show a
different dominant trend that may be buried among other patterns.

Following these considerations, we employ the following analytical pipeline:

  1. 1.

Calculate the $O$-information for multiplets of sizes from $3$ to $m$ (with
$m\leq n$ chosen considering computational convenience in case $n$ is too
large, or specific questions; the multiplet with n=2 represents pairwise
mutual information).

  2. 2.

Partition the multiplets depending on the sign of the O-information into
positive (redundant) and negative (synergistic).

  3. 3.

Evaluate the significance of the O-information values via bootstrap [18].

  4. 4.

Compare the bootstrap confidence interval [17] of each multiplet with subsets
of its components one order below. Multiplets are discarded if the confidence
intervals overlap, as this indicates that they do not display additional
synergy or redundancy.

  5. 5.

Build two hypergraphs with the significant multiplets: one with hyperedges of
positive O-information related to dominantly redundant interdependencies, and
another with hyperedges of negative O-inforamtion associated to synergy.

The output of this pipeline are redundancy and a synergy hypergraphs, which
give an overview of the structure of interdependecies in the system by
encoding the multiplets whose interaction are dominantly redundant or
synergistic, respectively. These two hypergraphs can then be used to perform
further analysis related to their topology, including centrality or clustering
analyses.

In this paper, step 1 is carried out via a Gaussian copula estimator, as
described in [31], which allows a parsimonious yet robust estimation of the
O-information. For step 3, the significance is evaluated by means of a
bootstrap procedure. The significant multiplets are those with a confidence
interval not containing zero and have an associated $p$-value smaller than a
fixed significance level, e.g. $0.01$), corrected for multiple comparisons
using the Bonferroni-Holm false discovery rate (FDR). After step 4 is
completed, the output will be a collection $C$ of multiplets of different
lengths, ${X_{1},\dots,X_{m}}$ with $m<n$, featuring any of the $n$ variables
under analysis. This collection $C$ can then be interpreted as a list of
hyperlinks indicating non-overlapping higher order interactions between the
considered variables. For illustration, the analyses presented in this paper
focus on multiplets of order 3, 4, and 5.

###  2.4 Generating higher-order effects on synthetic data

In order to test hypotheses pertaining to higher order properties, it is
useful to have the capability to generate synthetic data with specific levels
of higher order information. For this purpose, we employ a general approach
based on three steps, which are described as follows. First, we construct
triplets of variables generically named x,y,z with a pre-specified level of
higher order information — specifically, we focus on multivariate Gaussian
systems and build $3\times 3$ covariance matrices that exhibits a desired
level of higher order information. These covariance matrices are then stacked
on the diagonal of a $P\times P$ covariance matrix, where $P$ is the total
number of nodes in the system. As second step, we build connections between
some of these triplets (if desired) by setting the corresponding elements of
the off-diagonal covariance matrix to non-zero values. As final step, we use
the resulting covariance matrix to generate multivariate Gaussian data. To
reduce the corresponding degrees of freedom, we set the means to zero;
moreover as the scale is not important we standardize each of the variables —
so that the covariance matrix of the resulting data is in fact a correlation
matrix. The most challenging step of this process is the construction of
$3\times 3$ correlation matrices with a pre-specified level of higher order
information — for which we developed a numerical pipeline using the R package
lavaan [53], which is described in Appendix A.

Having discussed the basic elements of our procedure, we now present several
potential model variations, changing the level of interaction information for
one or more triplets. For this purpose, each of the suggested models has been
scaled up to include between nine and twelve triplets. All of the following
layouts will use one particular set of residual correlations, meaning that
only residual correlations are used between z-variables, or those only between
y\- and z-variables. The layout designs in figure 2 show how each node from a
triplet can be connected to two, three or four nodes originating from other
triplets.

Figure 2: Visual overview of the presented model layouts. 1) Nine independent
triplets. 2) Three clusters of three triplets. All triplets within a cluster
are connected to two others via their z-variables. 3) Expansion of Model
Layout 2. Three clusters of four triplets. Each triplet within a cluster is
connected to the three other related triplets via the z-variables.

The levels of the residual correlations per model correspond to the maximum
allowed in the case of forcing each triplet to include the maximum possible
amount of synergy within the model.

##  3 Results

###  3.1 Synthetic datasets

We first analysed a system composed of 27 variables arranged into nine
triplets with various covariance structures, which were — as triplets —
independent of each other. We considered three covariance structures, which
give zero, positive, and negative O-information, respectively (we informally
refer to the O-information of these triplets as “triplet information”). As
expected, our analysis pipeline correctly identifies redundant and synergistic
interdependencies in multiplets of order 3, but nothing in higher orders 3.
This provided a confirmation that our analysis pipeline is accurate in this
simple scenario.

Figure 3: Results obtained from Model 1 in Fig. 2 when implemented with zero
(i.e. independent), positive (i.e. redundant), and negative (i.e. synergistic)
interaction information in each triplet. Top row: network obtained with
$bootnet$ [19] and represented with $qgraph$ [epskamp2012qgraph], where which
both color intensity and edge thickness represent edge weight. Red indicates
negative edge weight, green indicates positive edge weight. Left: synergy;
Middle: Zero interaction information; Right: redundancy. Middle row:
significant multiplets of redundant $O$-information. Bottom row: significant
multiplets of synergistic $O$-information.

As a second step, we analysed scenarios where three groups of three triplets
each were linked through their z variables (see Figure 2). These links through
the $z$ variables in each triplet — weaker than the inter-triplet links —
resulted in the appearance of redundant multiplets in the independent case 4.
For the case of positive triplet information, these links result on new
redundant triplets connecting the $z$ variables of different triplets,
generating a more connected hypernetwork. The synergistic triplets are not
much disturbed by these links, while one multiplet of order 4 appears.
Interestingly, while the resulting hypergraph of redundant hyperedges follows
the pairwise structure, it is much less straightforward to derive the
resulting synergistic hyperedges.

Figure 4: Analogous results as shown in Figure 3 but for Model 2 in Figure 2.

Finally, we analysed scenarios with three groups of four triplets linked
through the z variables. The between-triplets links result in synergistic
multiplets of order 3 also in the independent and redundant cases. In the
synergistic case synergy extends to order 4 maintaining the same integration
of linked multiplets. Overall, the inclusion of larger cliques result in a
richer array of higher order interdependencies, resulting in redundant and
synergistic hypergraphs that depart more dramatically from the corresponding
pairwise network.

Figure 5: Analogous results as shown in Figure 3 but for Model 3 in Figure 2.

###  3.2 Reanalysis of the Empathy dataset — [11]

We re-analysed the psychometric data presented in [11] around the construct of
empathy ($N=1973$). We find (6) that the O-information can identify
significant synergistic and redundant interactions co-existing within the
hypergraph representation of the construct. Interestingly, multiplets of
orders 3 and 4 identify the same major factors obtained with the pairwise
psychometric network construction — namely fantasy, perspective taking,
personal distress and empathic concern. Since our network embeddings tend to
place at the center items engaging in more multiplets at once, our hypergraph
visualisations confirm Briganti and colleagues’ findings of empathic concern
being a pivotal dimension of empathy. However, this happens only when
redundant interactions up to order 3 and 4: at higher multiplet orders fantasy
becomes a more relevant major factor. This highlights higher order
interactions between cognitive-related projection abilities into fictional
characters, measuring fantasy, and more emotionally-focused items relative to
distress, empathy, and anticipation into the future. Please note that these
distinct trends could not have been retrieved by pairwise psychometric
networks.

Figure 6: Empathy dataset from [11]. Colors indicate membership of a component
of items, as in the reference paper. Top row: network obtained with $bootnet$.
Middle row: significant multiplets of redundant $O$-information. Bottom row:
significant multiplets of synergistic $O$-information.

These results agree with recent independent studies indicating a stronger-
than-expected interplay between fantasy and emotions in regulating empathy
[44]. Interestingly, synergistic interactions highlight items 3 and 24 as
pivotal. Whereas item 3 relates to the ability to see things from the
perspective of other individuals, and is thus evidently crucial for
synergistic interactions relative to empathy, item 24 (“I tend to lose control
during emergencies”) is relative to emotional regulation and distress. Our
results indicate that emotional regulation — the ability to exert control over
emotional responses — provides information over the whole construct when
combined with items of all other four factors.

The retrieved informational hyperedges provide relevant quantitative support
to other recent studies [65] showing how emotional regulation and empathy
greatly overlap, mainly in terms of understanding emotions, although at
different levels. Although the causal directionality between empathy and
emotion regulation has not been clearly assessed, exploratory studies have
shown how a lack of empathy can trigger emotional dysregulation in young
adults [56], further confirming the synergistic centrality of emotion
regulation found in our hypergraph representation.

###  3.3 Reanalysis of the PTSD dataset — [1]

As a second re-analysis, we studied the PTSD dataset [1] which includes
information about $N=221$, U.S. veterans. Our analyses did not identify
significant synergistic interactions between variables (7), which reflects a
lack of high-order interdependencies across items in the four major factors —
which include elements as distinct as intrusions, avoidance, cognitive and
mood alterations, and arousal and reactivity alterations. In contrast, several
redundant interactions where found among items in multiplets of orders 3 and
4, which suggest that these factors may be driven by similar underlying
factors. These redundant O-information hyperedges identifies items about
nightmares, flashbacks, physiological cue reactivity and avoidance of
reminders as being highly central. This combination of behavioural, emotional
and memory-related patterns underlines the complexity of PTSD as a condition
affecting not only the emotional and cognitive spheres of individuals, but
also their behaviour and physiology [2].

Figure 7: PTSD dataset from [1]. Colors indicate membership of a class of
items. Top row: network obtained with $bootnet$. Middle row: significant
multiplets of redundant $O$-information. Bottom row: significant multiplets of
synergistic $O$-information.

The lack of a clearer internal structure and of synergistic interactions might
be due not only to an intrinsic complexity of the construct, but also to
sample size issues, missing item in scales and other methodological issues. To
better understand this, we re-analysed the same dataset while using a larger
number of variables [1] to include covariates (8). In this second hypergraph,
clinical covariates like anxiety and depression levels were identified as
being pivotal for redundant interactions across multiplet orders 3, 4 and 5.
This indicates that anxiety and depression levels overlap significantly with
behavioural, emotional and physiological symptoms of PSTD, in agreement with
independent studies [2]. The addition of clinical covariates polarised the
hypergraph structure and led to a more pronounced separation between intrusion
variables and others. Furthermore, intrusive thoughts were found to be an item
bridging multiplets at order 4 in terms of redundant information. Hence, our
analysis indicates a key role mediated by intrusive roles as possessing richly
redundant information and thus being relevant for understanding the construct.

Figure 8: PTSD dataset from [1]. Colors indicate membership of a class of
items. Grey items are the covariates. Top row: network obtained with
$bootnet$. Middle row: significant multiplets of redundant $O$-information.
Bottom row: significant multiplets of synergistic $O$-information.

Our findings agree with other studies highlighting intrusive thoughts as key
signs of PTSD [13], potentially rising from a maladaptive thought suppression,
i.e. a conscious, imperfect, cognitive-driven suppression of a negative
thought can enhance the prominence and recurrence of the thought itself,
causing worry and rumination. The persistence of intrusive thoughts has
serious documented consequences over behaviour and emotional wellbeing [13,
2], which support the redundancy higher order patterns found in our work.
Interestingly, synergistic interactions miss even in this expanded set of
items. Future research should investigate whether this pattern is due to
sample size issues or other methodological aspects of PTSD measurements.

##  4 Discussion

We have put forward a novel approach to capture higher order interactions
between observables in psychometrics and described them in the form of
hypergraphs. In particular, we introduced an analysis pipeline that builds two
hypegraphs reflecting redundant and synergistic interdependencies. This
approach capitalizes on recent advances in multivariate information theory,
which have provided data-efficient metrics to capture these phenomena in
multiplets of different sizes, and has the scope to build up and extend
existent network approaches to psychometric.

Current network approaches to psychometric are based on a key assumption: that
relationships between variables can most of the time be represented as
pairwise networks — either in cases of having a common cause or effect, or
being linked by a directed flow of influences (i.e. as a directed acyclic
graph) or a Ising-like structure [10]. This assumption, in turn, leads to the
adoption of pairwise random Markov fields as a natural modelling choice for
identifying the interactions between variables. However, recent work [73, 41,
27] has convincingly shown the relevance of higher order interdependencies, in
which relationships often take place between groups of variables in a way that
cannot be reduced to a set of pairwise links. Building on these insights, our
proposed framework aims to relax these assumptions along two axes.

On the first axis, we propose to disentangle the analysis of high order
mechanisms (and structures) from the high order behaviors that can emerge in
complex systems. As described in [50] in the context of physical systems,
there is a fundamental distinction between mechanisms that address how the
system is structured, and behaviors that correspond to emergent properties
related to what the system “does.” Furthermore, the existence of higher order
mechanisms do not imply high order behaviors or vice-versa. Consequently, our
proposed approach focuses on the analysis of high order patterns in the data,
while explicitly acknowledging that this does not bring straightforward
implications on how such data was generated in the first place. This angle
also underlines the main difference with another way of seeing higher-order
networks in psychometrics, where the links involve latent factors considered
as mediators in three-way relationships evidenced by mixed graphical models
[29] 111To stress where this approach is different, consider a simple linear
regression $Y\sim X1+X2$. It could be that the triplet $(Y,X1,X2)$ exhibits
redundancy, synergy or independence. According to [29], a second-order model
would be $Y\sim X1+X2+X1*X2$, involving the addition of more
information/variables (product terms). This second-order model would need a
4x4 covariance matrix, while our approach would stick to a 3x3 covariance
matrix. On the other hand one could, of course, create a quadruplet (Y, X1,
X2, X1*X2) that exhibits redundancy or synergy. In other words, the two
definitions are complementary, and they can be combined, but they refer to
different phenomena..

The second axis relates to possible confounding effects, and their nature.
Variables sharing information, yet considered as individual or pairwise
interacting units, can pose problems for proper network reconstruction and in
assigning a given centrality score to an individual variable. The presence of
a variable could either enhance or deteriorate statistical dependencies
between other variables, possibly influencing results for network estimation
when using established centrality indices of strength, betweenness, and
closeness [28]. Analogous issues of deteriorated model validity would hold
also for other approaches like factor analysis [66]. Another possibility could
be the existence of a statistical dependency between two variables which is
considered insignificant in an exploratory analysis [25, 19]. This issue
increases the concern of detecting false positives regarding statistical
dependencies during estimation. While some efforts have been done in this
direction (e.g. [29]), correlations can persist even after conditioning on a
latent variable, a phenomenon known as local dependency [39]. The persistence
of correlated residuals can originate from multiple causes [24, 39, 54, 71],
e.g. by construction (scale items are selected as already being strongly
correlated with each other), because of variables sharing semantic framing (in
their item descriptions), or because of a lack of randomisation in the order
of items and subsequent cross-item contextual assimilation effects. Local
dependency violates the assumptions of commonly used psychometric models of
latent variables, like Item Response Theory or factor analysis [39]. If
significant correlations persist between items even after the contribution of
the latent variable is removed, then items are locally dependent and this
creates unaccounted model dimensions, with consequences like factor loading
inflation or factor splitting in noisy/theoretically unexpected sets of
variables. Advantageous solutions to overcome local independence are either to
bundle sets of locally dependent items into polytomous super-items [39, 54] in
the setting where the items are binary, or consider 3-way interactions [29].
Both these advancements pose novel challenges about preserving statistical,
diagnostic and interpretable measures. Our work contributes quantitatively to
this direction while exploiting the powerful statistical framework of higher
order interactions in complex networks [50]. Considering higher order
interactions involving the observables only, and thus transferring these
directly to hyperedges may let network approaches even more complementary to
classical factor analysis.

Our approach may also be helpful for detecting latent factors and correctly
attribute observables to them, which is also the goal of recent work aimed at
modelling local dependency as a trace for redundancy [15, 26, 24, 70].
Redundancy comes from the fact that within a latent variable model, locally
dependent variables contain less information than the one predicted by the
(inflated) model parameters. Christensen and colleagues [24] measured
redundancy and local dependency by considering psychometric networks of
pairwise relationships enriched with weighted topological overlap, i.e. a
metric estimating how similarly connected are any two nodes in a network.
Their innovative approach showed how considering “higher order” interactions
through network overlap provided superior factor estimation compared to IRT
modelling. Our approach adopts a different perspective, substituting the
pairwise psychometric network with a hypergraph structure in which higher
order interactions are naturally captured by information-theoretic hyperlinks.
Thus, it is worth noting that the terms “higher order,” “redundancy” or “3-way
interactions” can have a different flavor or even some different definitions
than the ones used so far. An important difference is that the structures
simulated in the papers cited above do indeed generate redundancy (in the
sense of high order informational content), but not synergy (see B). Whereas
redundancy can be captured by local dependency between variables, synergy must
be interpreted in different terms [61], i.e. as the amount of information
available only when variables are jointly considered. Our approach enables the
quantification of synergistic interactions also within psychometric networks,
extending to psychological representations a measure originally devised for
brain data and networks.

The presented framework uses the O-information in its basic form, but there is
a number of variations of it which can be used for future work. In effect,
recent work have introduced multiple extensions of the O-information,
including dynamical versions [63] that provide an expansion of Transfer
Entropy (or Granger Causality in the Gaussian case [4]), point-wise extensions
which provide O-information values for each individual pattern [55], and a
spectral decomposition that resolve the O-information into different frequency
bands [21]. These tools could be used to build hypergraphs extending temporal
psychological networks, as those presented in [22]. Please note that the
metrics introduced in [63] can be used to calculate conditional O-information
metrics, which provide a close simile to the conditional correlation that is
often used in the network psychometric literature. Other interesting avenues
of interesting future work include leveraging recent algorithms to identify
hypergraph centrality [67] and modularity [35, 14], and also harmonic analysis
and dimensionality reduction [42].

One limitation of the O-information (and any other coarse-grained metric over
an information decomposition [74]) is that a multiplet is labelled as
synergistic or redundant depending on its sign, and in this sense it reflects
the net balance between the two quantities and only acknowledges the dominant
one. Note that redundancy might originate from underlying psychological
correlations, for instance two or more symptoms might overlap rather than
causing one another, thus bringing more redundancy within the analysis [19].
For these reasons redundancy is prevalent in real datasets — where it is most
likely to be found by experimental design or construction, and hence this
could cover synergistic interactions making them harder to find. One solution
could be to first reduce the redundancy in the sense of common factors [24],
or consider implementing a full partial information decomposition (e.g. as
proposed in [7, 33, 51]) where redundancy and synergy can be considered as
distinct information atoms that can be assessed entirely separately.

Another important open question is how to best visualize hypergraphs. It is
known how pairwise networks can be elegant and visually inspiring, but when
overcrowded they can become hard to interpret — resembling an spaghetti ball.
In hypergraphs this visualization-level curse of dimensionality appears
earlier, and we acknowledge that some of the plots presented here (made
possible thanks to HyperNetX [45]) can give inspiration for crop circles
design. This is not a valid reason to escape the fact that networks are
complicated and overcrowded, and hopefully new designs and specialised
software will come up.

To conclude, by taking inspiration in complex system whose behavior is not
equal to the one resulting from sum of its parts, here we asked ourselves
whether the interdependencies observed in psychometric data could be described
in a way that could account for beyond-pairwise (albeit conditioned), higher
order interactions. Our proposed framework builds hyperlinks directly from the
multivariate distribution of the data, identifying multiplets with a
dominantly redundant or synergistic role. These analyses may ultimately inform
on _which kind_ of information is being shared by different behavioral
variables or symptoms, and how these informational multiplets change in time.
This higher-order perspective complements current network approaches, leading
to ways of assessing the data that greatly differ from what can be attained by
other methods such as factor analyses. By doing this, the proposed approach
enriches the toolbox of psychometricians and opens promising avenues for
future investigation.

##  5 Data and code availability

The empirical data are found at these repositories https://osf.io/73c4q/ and
https://osf.io/mj5wa/. The code for simulations, analyses, and plotting is
described in this github repository
https://github.com/danielemarinazzo/O_info_psycnet.

##  6 Acknowledgements

F.R. was supported by the Ad Astra Chandaria foundation. M.S. acknowledges
Luigi Lombardi for discussions. R.C. was supported by the Mind, Brain and
Reasoning PhD programme at the University of Milan. S.S. was supported by MIUR
project PRIN 2017WZFTZP, “Stochastic forecasting in complex systems.”

## References

  * [1] Cherie Armour et al.  “A network analysis of DSM-5 posttraumatic stress disorder symptoms and correlates in U.S. military veterans”  In _Journal of Anxiety Disorders_ 45, 2017, pp. 49–59  DOI: https://doi.org/10.1016/j.janxdis.2016.11.008
  * [2] Cherie Armour et al.  “Posttraumatic stress disorder symptoms and reckless behaviors: A network analysis approach”  In _Journal of Traumatic Stress_ 33.1 Wiley Online Library, 2020, pp. 29–40
  * [3] Eyal Bairey, Eric D Kelsic and Roy Kishony  “High-order species interactions shape ecosystem diversity”  In _Nature communications_ 7.1 Nature Publishing Group, 2016, pp. 1–7
  * [4] Lionel Barnett, Adam B Barrett and Anil K Seth  “Granger causality and transfer entropy are equivalent for Gaussian variables”  In _Physical review letters_ 103.23 APS, 2009, pp. 238701
  * [5] Federico Battiston et al.  “The physics of higher-order interactions in complex systems”  In _Nature Physics_ 17.10 Nature Publishing Group, 2021, pp. 1093–1098
  * [6] Federico Battiston et al.  “Networks beyond pairwise interactions: structure and dynamics”  In _Physics Reports_ 874 Elsevier, 2020, pp. 1–92
  * [7] Nils Bertschinger et al.  “Quantifying unique information”  In _Entropy_ 16.4 Multidisciplinary Digital Publishing Institute, 2014, pp. 2161–2183
  * [8] Nitin Bhushan et al.  “Using a Gaussian graphical model to explore relationships between items and variables in environmental psychology research”  In _Frontiers in psychology_ 10 Frontiers, 2019, pp. 1050
  * [9] Denny Borsboom and Angélique OJ Cramer  “Network analysis: an integrative approach to the structure of psychopathology”  In _Annual review of clinical psychology_ 9 Annual Reviews, 2013, pp. 91–121
  * [10] Denny Borsboom et al.  “Network analysis of multivariate data in psychological science”  In _Nature Reviews Methods Primers_ 1.1 Nature Publishing Group, 2021, pp. 1–18
  * [11] Giovanni Briganti et al.  “Network analysis of empathy items from the interpersonal reactivity index in 1973 young adults”  In _Psychiatry Research_ 265 Elsevier, 2018, pp. 87–92
  * [12] Laura F Bringmann et al.  “What do centrality measures measure in psychological networks?”  In _Journal of Abnormal Psychology_ 128.8 American Psychological Association, 2019, pp. 892
  * [13] Matthew Brooks, Nicola Graham-Kevan, Sarita Robinson and Michelle Lowe  “Trauma characteristics and posttraumatic growth: The mediating role of avoidance coping, intrusive thoughts, and social support.”  In _Psychological Trauma: Theory, Research, Practice, and Policy_ 11.2 Educational Publishing Foundation, 2019, pp. 232
  * [14] Philip S Chodrow, Nate Veldt and Austin R Benson  “Generative hypergraph clustering: From blockmodels to modularity”  In _Science Advances_ 7.28 American Association for the Advancement of Science, 2021, pp. eabh1303
  * [15] Alexander P Christensen, Luis Eduardo Garrido and Hudson F Golino  “Unique variable analysis: A novel approach for detecting redundant variables in multivariate data”  PsyArXiv, 2020
  * [16] Angélique OJ Cramer, Lourens J Waldorp, Han LJ Van Der Maas and Denny Borsboom  “Comorbidity: A network perspective”  In _Behavioral and brain sciences_ 33.2-3 Cambridge University Press, 2010, pp. 137–150
  * [17] Thomas J DiCiccio and Bradley Efron  “Bootstrap confidence intervals”  In _Statistical science_ 11.3 Institute of Mathematical Statistics, 1996, pp. 189–228
  * [18] Bradley Efron and Robert J Tibshirani  “An introduction to the bootstrap”  CRC press, 1994
  * [19] Sacha Epskamp, Denny Borsboom and Eiko I Fried  “Estimating psychological networks and their accuracy: A tutorial paper”  In _Behavior Research Methods_ 50.1 Springer, 2018, pp. 195–212
  * [20] Sacha Epskamp and Eiko I Fried  “A tutorial on regularized partial correlation networks.”  In _Psychological methods_ 23.4 American Psychological Association, 2018, pp. 617
  * [21] Luca Faes et al.  “A Framework for the Time-and Frequency-Domain Assessment of High-Order Interactions in Brain and Physiological Networks”  In _arXiv preprint arXiv:2202.04179_ , 2022
  * [22] Eiko I Fried, Faidra Papanikolaou and Sacha Epskamp  “Mental health and social contact during the COVID-19 pandemic: an ecological momentary assessment study”  In _Clinical Psychological Science_ 10.2 SAGE Publications Sage CA: Los Angeles, CA, 2022, pp. 340–354
  * [23] Eiko I Fried et al.  “Mental disorders as networks of problems: a review of recent insights”  In _Social Psychiatry and Psychiatric Epidemiology_ 52.1 Springer, 2017, pp. 1–10
  * [24] Hudson F Golino et al.  “Modeling latent topics in social media using Dynamic Exploratory Graph Analysis: The case of the right-wing and left-wing trolls in the 2016 US elections”  In _Psychometrika_ Springer, 2021, pp. 1–32
  * [25] Hudson F Golino and Sacha Epskamp  “Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research”  In _PloS One_ 12.6 Public Library of Science San Francisco, CA USA, 2017, pp. e0174035
  * [26] Hudson F Golino et al.  “Entropy fit indices: New fit measures for assessing the structure and dimensionality of multiple latent variables”  In _Multivariate Behavioral Research_ 56.6 Taylor & Francis, 2021, pp. 874–902
  * [27] Matteo Grasso, Larissa Albantakis, Jonathan P Lang and Giulio Tononi  “Causal reductionism and causal structures”  In _Nature neuroscience_ 24.10 Nature Publishing Group, 2021, pp. 1348–1355
  * [28] Michael N Hallquist, Aidan GC Wright and Peter CM Molenaar  “Problems with centrality measures in psychopathology symptom networks: Why network psychometrics cannot escape psychometric theory”  In _Multivariate Behavioral Research_ 56.2 Taylor & Francis, 2021, pp. 199–223
  * [29] Jonas MB Haslbeck, Denny Borsboom and Lourens J Waldorp  “Moderated network models”  In _Multivariate behavioral research_ 56.2 Taylor & Francis, 2021, pp. 256–287
  * [30] Iacopo Iacopini, Giovanni Petri, Alain Barrat and Vito Latora  “Simplicial models of social contagion”  In _Nature communications_ 10.1 Nature Publishing Group, 2019, pp. 1–9
  * [31] Robin A.A. Ince et al.  “A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula”  In _Human Brain Mapping_ 38.3, 2017, pp. 1541–1573  DOI: https://doi.org/10.1002/hbm.23471
  * [32] Ryan G James and James P Crutchfield  “Multivariate dependence beyond Shannon information”  In _Entropy_ 19.10 Multidisciplinary Digital Publishing Institute, 2017, pp. 531
  * [33] Ryan G James, Jeffrey Emenheiser and James P Crutchfield  “Unique information via dependency constraints”  In _Journal of Physics A: Mathematical and Theoretical_ 52.1 IOP Publishing, 2018, pp. 014002
  * [34] Payton J Jones, Ruofan Ma and Richard J McNally  “Bridge centrality: A network approach to understanding comorbidity”  In _Multivariate behavioral research_ Taylor & Francis, 2019, pp. 1–15
  * [35] Bogumił Kamiński, Paweł Prałat and François Théberge  “Community detection algorithm using hypergraph modularity”  In _International Conference on Complex Networks and Their Applications_ , 2020, pp. 152–163  Springer
  * [36] Peter F Lovibond and Sydney H Lovibond  “The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories”  In _Behaviour research and therapy_ 33.3 Elsevier, 1995, pp. 335–343
  * [37] David J. Marchette  “HyperG: Hypergraphs in R” R package version 1.0.0, 2021  URL: https://CRAN.R-project.org/package=HyperG
  * [38] M Marsman et al.  “An introduction to network psychometrics: Relating Ising network models to item response theory models”  In _Multivariate behavioral research_ 53.1 Taylor & Francis, 2018, pp. 15–35
  * [39] Roderick P McDonald  “Test theory: A unified treatment”  psychology press, 2013
  * [40] William McGill  “Multivariate information transmission”  In _Transactions of the IRE Professional Group on Information Theory_ 4.4 IEEE, 1954, pp. 93–111
  * [41] Pedro AM Mediano et al.  “Towards an extended taxonomy of information dynamics via Integrated Information Decomposition”  In _arXiv preprint arXiv:2109.13186_ , 2021
  * [42] Anibal M Medina-Mardones, Fernando E Rosas, S E Rodriguez and Rodrigo Cofre  “Hyperharmonic analysis for the study of high-order information-theoretic signals”  In _Journal of Physics: Complexity_ 2.3 IOP Publishing, 2021, pp. 035009
  * [43] Staša Milojević  “Principles of scientific research team formation and evolution”  In _Proceedings of the National Academy of Sciences_ 111.11 National Acad Sciences, 2014, pp. 3984–3989
  * [44] Kohei Nomura and Seiki Akai  “Empathy with fictional stories: reconsideration of the fantasy scale of the interpersonal reactivity index”  In _Psychological Reports_ 110.1 SAGE Publications Sage CA: Los Angeles, CA, 2012, pp. 304–314
  * [45] B Praggastis et al.  “HyperNetX”  In _GitHub repository_ GitHub, https://github.com/pnnl/HyperNetX, 2019
  * [46] Julia M Rohrer  “Thinking clearly about correlations and causation: Graphical causal models for observational data”  In _Advances in methods and practices in psychological science_ 1.1 Sage Publications Sage CA: Los Angeles, CA, 2018, pp. 27–42
  * [47] Fernando E Rosas et al.  “Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data”  In _PLoS computational biology_ 16.12 Public Library of Science San Francisco, CA USA, 2020, pp. e1008289
  * [48] Fernando E Rosas, Pedro Mediano, Martı́n Ugarte and Henrik J Jensen  “An information-theoretic approach to self-organisation: Emergence of complex interdependencies in coupled dynamical systems”  In _Entropy_ 20.10 Multidisciplinary Digital Publishing Institute, 2018, pp. 793
  * [49] Fernando E Rosas, Pedro A.. Mediano, Michael Gastpar and Henrik J. Jensen  “Quantifying high-order interdependencies via multivariate extensions of the mutual information”  In _Phys. Rev. E_ 100 American Physical Society, 2019, pp. 032305  DOI: 10.1103/PhysRevE.100.032305
  * [50] Fernando E Rosas et al.  “Disentangling high-order mechanisms and high-order behaviours in complex systems”  In _Nature Physics_ Nature Publishing Group, 2022, pp. 1–2
  * [51] Fernando E Rosas, Pedro AM Mediano, Borzoo Rassouli and Adam B Barrett  “An operational information decomposition via synergistic disclosure”  In _Journal of Physics A: Mathematical and Theoretical_ 53.48 IOP Publishing, 2020, pp. 485001
  * [52] Fernando E Rosas et al.  “Understanding interdependency through complex information sharing”  In _Entropy_ 18.2 Multidisciplinary Digital Publishing Institute, 2016, pp. 38
  * [53] Yves Rosseel  “Lavaan: An R package for structural equation modeling and more. Version 0.5–12 (BETA)”  In _Journal of statistical software_ 48.2, 2012, pp. 1–36
  * [54] John Rust and Susan Golombok  “Modern psychometrics: The science of psychological assessment”  Routledge, 2014
  * [55] Tomas Scagliarini et al.  “Quantifying high-order interdependencies on individual patterns via the local O-information: theory and applications to music analysis”  In _Physical Review Research_ 4.1 APS, 2022, pp. 013184
  * [56] Marc Schipper and Franz Petermann  “Relating empathy and emotion regulation: Do deficits in empathy trigger emotion dysregulation?”  In _Social neuroscience_ 8.1 Taylor & Francis, 2013, pp. 101–107
  * [57] Verena D Schmittmann et al.  “Deconstructing the construct: A network perspective on psychological phenomena”  In _New ideas in psychology_ 31.1 Elsevier, 2013, pp. 43–53
  * [58] Claude E Shannon  “A mathematical theory of communication”  In _Bell system technical journal_ 27.3 Wiley Online Library, 1948, pp. 379–423
  * [59] Per Sebastian Skardal and Alex Arenas  “Higher order interactions in complex networks of phase oscillators promote abrupt synchronization switching”  In _Communications Physics_ 3.1 Nature Publishing Group, 2020, pp. 1–6
  * [60] Guillaume St-Onge et al.  “Influential groups for seeding and sustaining nonlinear contagion in heterogeneous hypergraphs”  In _Communications Physics_ 5.1 Nature Publishing Group, 2022, pp. 1–16
  * [61] Sebastiano Stramaglia et al.  “Synergetic and redundant information flow detected by unnormalized Granger causality: Application to resting state fMRI”  In _IEEE Transactions on Biomedical Engineering_ 63.12 IEEE, 2016, pp. 2518–2524
  * [62] Sebastiano Stramaglia, Jesus M Cortes and Daniele Marinazzo  “Synergy and redundancy in the Granger causal analysis of dynamical networks”  In _New Journal of Physics_ 16.10 IOP Publishing, 2014, pp. 105003
  * [63] Sebastiano Stramaglia, Tomas Scagliarini, Bryan C Daniels and Daniele Marinazzo  “Quantifying dynamical high-order interdependencies from the o-information: an application to neural spiking dynamics”  In _Frontiers in Physiology_ Frontiers, 2021, pp. 1784
  * [64] Han Te Sun  “Nonnegative entropy measures of multivariate symmetric correlations” Publisher: Elsevier  In _Information and Control_ 36, 1978, pp. 133–156
  * [65] Nicholas M Thompson, Andero Uusberg, James J Gross and Bhismadev Chakrabarti  “Empathy and emotion regulation: An integrative account”  In _Progress in brain research_ 247 Elsevier, 2019, pp. 273–304
  * [66] Daniel E Tuccitto, Peter R Giacobbi Jr and Walter L Leite  “The internal structure of positive and negative affect: A confirmatory factor analysis of the PANAS”  In _Educational and psychological measurement_ 70.1 Sage Publications Sage CA: Los Angeles, CA, 2010, pp. 125–141
  * [67] Francesco Tudisco and Desmond J Higham  “Node and edge nonlinear eigenvector centrality for hypergraphs”  In _Communications Physics_ 4.1 Nature Publishing Group, 2021, pp. 1–10
  * [68] Nathan Van den Bergh, Igor Marchetti and Ernst HW Koster  “Bridges over troubled waters: Mapping the interplay between anxiety, depression and stress through network analysis of the DASS-21”  In _Cognitive Therapy and Research_ 45.1 Springer, 2021, pp. 46–60
  * [69] Vaiva Vasiliauskaite and Fernando E Rosas  “Understanding complexity via network theory: a gentle introduction”  In _arXiv preprint arXiv:2004.14845_ , 2020
  * [70] Ranjith Vijayakumar, Ji Yeh Choi and Eun Hwa Jung  “A Unified Neural Network Framework for Extended Redundancy Analysis”  In _Psychometrika_ , 2022  DOI: 10.1007/s11336-022-09853-x
  * [71] Wen-Chung Wang and Mark Wilson  “Exploring local item dependence using a random-effects facet model”  In _Applied Psychological Measurement_ 29.4 Sage Publications Sage CA: Thousand Oaks, CA, 2005, pp. 296–318
  * [72] Satosi Watanabe  “Information theoretical analysis of multivariate correlation” Publisher: IBM  In _IBM Journal of research and development_ 4.1, 1960, pp. 66–82
  * [73] Paul L Williams  “Information dynamics: Its theory and application to embodied cognitive systems”, 2011
  * [74] Paul L Williams and Randall D Beer  “Nonnegative decomposition of multivariate information”  In _arXiv preprint arXiv:1004.2515_ , 2010

##  Appendix A Simulation of synergy and redundancy with lavaan

From Eqs. (3), (1), and (2), the O-information between three variables
$(X,Y,Z)$ can be calculated as

$\Omega(X;Y;Z)=I(X;Y)-I(X;Y|Z).$ (4)

Moreover, if we assume $(X,Y,Z)$ follow a multivariate Gaussian distribution,
then their O-information can be calculated from their correlation matrix as
follows:

$\Omega(X;Y;Z)=\frac{1}{2}\log\frac{1-\rho(X,Y|Z)^{2}}{1-\rho(X,Y)^{2}}=\frac{1}{2}\log\frac{|\Sigma|}{|\Sigma_{xy}\Sigma_{yz}\Sigma_{xz}|},$
(5)

Above, the first equality uses the fact that for multivariate Gaussian
variables $I(X;Y)=-\frac{1}{2}\log(1-\rho(X,Y)^{2})$ and
$I(X;Y|Z)=-\frac{1}{2}\log(1-\rho(X,Y|Z)^{2})$, where $\rho(X,Y)$ is the
Pearson correlation between $X$ and $Y$ and $\rho(X,Y|Z)$ is the conditional
Pearson correlation of $X$ and $Y$ given $Z$, which can be calculated as

$\rho(X,Y|Z)=\frac{\rho(X,Y)-\rho(X,Z)\rho(Y,Z)}{\sqrt{1-\rho(X,Z)^{2}}\sqrt{1-\rho(Y,Z)^{2}}}.$
(6)

The second equality can be directly verified, and denotes as $|\Sigma|$ the
determinant of the correlation matrix of $(X,Y,Z)$ and $|\Sigma_{ij}|$ the
determinant of the submatrix of the corresponding variables.

Overall, the second expression shows that the O-information is transformation
invariant (i.e. it does not depends on the mean values or variances of its
arguments but only on the three correlations $\rho_{xy}$, $\rho_{yz}$, and
$\rho_{xz}$), and the latter that it is symmetric on its three arguments.

While finding the interaction information that corresponds to three
correlation values is straightforward, the reverse operation information is
highly non-trivial. To facilitate this process, we will use a parameterization
based on factor analysis. The factor model with a single factor and $n$
indicators $y_{k}$, $k=1,2,\ldots,n$, can we written as

$y_{k}=\lambda_{k}\eta+\nu_{k}$ (7)

where $\eta$ is a latent random variable, $\lambda_{k}\in\mathbb{R}$ is the
factor loading for indicator $k$, and $\nu$ is additive Gaussian noise with
covariance matrix $\Theta$ with components $\theta_{i,j}$. For simplicity we
set $\text{Var}(\eta)=1$, $\text{E}(\theta_{k})=0$, and
$\text{Cov}(\eta,\theta_{k})=0$ for all $k=1,\dots,n$. Then it follows that
the covariance matrix of $y_{1},\dots,y_{n}$, denoted by $\Sigma$, is given by

$\Sigma=\lambda\lambda^{T}+\Theta,$ (8)

where $\lambda=(\lambda_{1},\dots,\lambda_{n})$.

If we have a triplet of three variables $X$, $Y$ and $Z$, the factor model can
be expressed by the equations:

$\displaystyle X=\lambda_{x}\eta+\theta_{x}$ $\displaystyle
Y=\lambda_{y}\eta+\theta_{y}$ $\displaystyle Z=\lambda_{z}\eta+\theta_{z}.$

In order to create a correlation matrix which either exhibits redundancy,
independence or synergy, we will first fix the factor loadings to apriori
chosen constants. Several choices are possible, but we have used
$\lambda_{x}=\sqrt{0.99}$, $\lambda_{y}=\sqrt{0.70}$ and
$\lambda_{z}=\sqrt{0.30}$. To ensure that the resulting covariance matrix has
a unit diagonal, we set the value of $\theta_{x}$, $\theta_{y}$ and
$\theta_{z}$ to $(1-0.99)$, $(1-0.70)$ and $(1-0.30)$ respectively. The only
parameter that we will vary is the error covariance (ecov)
$\text{Cov}(\theta_{y},\theta_{z})$. All other off-diagonal elements of
$\bm{\Theta}$ are fixed to zero. The possible range of this ecov parameter is
about $[-0.458,+0.458]$ as determined by $\sqrt{1-0.70}\times\sqrt{1-0.30}$.
Otherwise, the correlation matrix would no longer be positive definite.
Changing this ecov parameter allows us to set the level of interaction
information. For example, if we let ecov to be $-0.14849$, the level of
interaction information is smaller than 1e-07, implying independence. If we
let ecov to be $-0.39$ or $0.22$, the level of information is about $0.58$
(indicating synergy) and $-0.175$ (indicating redundancy) respectively. The
corresponding correlation matrices are shown in Table 3, Table 3 and Table 3,
where we use $A.x$, $A.y$ and $A.z$ to refer to the three variables that make
up the triplet $A$.

A.x A.y A.z A.x 1 A.y 0.832 1 A.z 0.545 0.310 1 Table 1: Zero interaction,
with $\textit{ecov}=-0.14849$ A.x A.y A.z A.x 1 A.y 0.832 1 A.z 0.545 0.068 1
Table 2: Synergy, with
$\textit{ecov}=-0.39$ A.x A.y A.z A.x 1 A.y 0.832 1 A.z 0.545 0.678 1 Table 3:
Redundancy, with $\textit{ecov}=0.22$

Obviously, the level of interaction information will have an impact on the
partial correlations. As a result, if we apply the graphical lasso to these
triplets and visualize them using the qgraph package, they result in very
different patterns as can be seen in Figure 9.

Figure 9: qgraph visualizations of the triplet. Both color intensity and edge
thickness represent edge weight. Red indicates negative edge weight, green
indicates positive edge weight. Left: synergy; Middle: Zero interaction
information; Right: redundancy.

A final model can combine several triplets, each with its own value for the
ecov parameter. The full covariance matrix is then constructed by placing the
model-implied covariance (correlation) matrices of the triplets on the main
diagonal. In addition, we can allow for variables to be connected by setting
the corresponding off-diagonal elements of the covariance matrix to a non-zero
value. An example covariance matrix for three connected triplets is shown in
Table 4.

| A.x | A.y | A.z | B.x | B.y | B.z | C.x | C.y | C.z
---|---|---|---|---|---|---|---|---|---
A.x | 1 |  |  |  |  |  |  |  |
A.y | 0.832 | 1 |  |  |  |  |  |  |
A.z | 0.545 | 0.678 | 1 |  |  |  |  |  |
B.x | 0 | 0 | 0 | 1 |  |  |  |  |
B.y | 0 | 0 | 0 | 0.832 | 1 |  |  |  |
B.z | 0 | 0 | 0.150 | 0.545 | 0.308 | 1 |  |  |
C.x | 0 | 0 | 0 | 0 | 0 | 0 | 1 |  |
C.y | 0 | 0 | 0 | 0 | 0 | 0 | 0.832 | 1 |
C.z | 0 | 0 | 0.150 | 0 | 0 | 0.150 | 0.545 | 0.068 | 1
Table 4: Covariance matrix representation of the example displayed in Figure
10 including off-diagonal covariances in bold.

A visualization of this covariance matrix is presented in Figure 10

Figure 10: qgraph network visualization of three triplets with residual
correlations between triplets. Triplet A includes redundancy; triplet B
includes zero interaction information; triplet C includes synergy. The primary
link between the triplets are done via the variables A.z, B.z and C.z.
Residual correlations were set at 0.15.

##  Appendix B Reanalysis of the Golino redundancy framework

Here we analyze the datasets simulated in [15, 26, 24] with 2000 observations,
three latent factors (factor loadings $=0.4$ and three variables per latent
factor, without ($c=0$, 11) and with ($c=0.3$, 12) correlation between
factors. Only redundancy emerges from the analysis with our framework.

Figure 11: Significant redundant multiplets when the factors are not
correlated. Figure 12: Significant redundant multiplets when the factors are
correlated.