# Time-series-analysis-based detection of critical transitions in real-world
non-autonomous systems

Klaus Lehnertz<EMAIL_ADDRESS>Department of Epileptology,
University of Bonn Medical Centre, Venusberg Campus 1, 53127 Bonn, Germany
Helmholtz Institute for Radiation and Nuclear Physics, University of Bonn,
Nussallee 14–16, 53115 Bonn, Germany  Interdisciplinary Center for Complex
Systems, University of Bonn, Brühler Straße 7, 53175 Bonn, Germany

###### Abstract

Real-world non-autonomous systems are open, out-of-equilibrium systems that
evolve in and are driven by temporally varying environments. Such systems can
show multiple timescale and transient dynamics together with transitions to
very different and, at times, even disastrous dynamical regimes. Since such
critical transitions disrupt the systems’ intended or desired functionality,
it is crucial to understand the underlying mechanisms, to identify precursors
of such transitions and to reliably detect them in time series of suitable
system observables to enable forecasts. This review critically assesses the
various steps of investigation involved in time-series-analysis-based
detection of critical transitions in real-world non-autonomous systems: from
the data recording to evaluating the reliability of offline and online
detections. It will highlight pros and cons to stimulate further developments,
which would be necessary to advance understanding and forecasting nonlinear
behavior such as critical transitions in complex systems.

> Critical transitions are apparent sudden shifts in the states of complex
> dynamical systems and are often followed by extreme events that may entail
> loss of life and/or materials. Reliable detection and forecasting of
> critical transitions would allow for developing adaptation and/or mitigation
> strategies, which is thus of utmost importance in many scientific fields.
> Although we still lack a general understanding of the mechanisms underlying
> critical transitions, there is plenty of observational data which might
> qualify for a time-series-analysis-based detection of transitions in real-
> world non-autonomous systems. Successful detection and forecasting of
> critical transitions requires not only knowledge of the various concepts and
> methodologies involved, but also an assessment of possible sources of error.
> This is the subject of the present review.

##  I Introduction

Complex dynamical systems, ranging from the brain and the heart to ecosystems,
financial markets and the earth and climate systems can show sudden
transitions to very different and, at times, even disastrous dynamical regimes
Albeverio, Jentsch, and Kantz (2006); Basher (2006); Willoughby, Rappaport,
and Marks (2007); Lenton (2011); Ansmann  _et al._ (2013); Meron (2015);
Trefois  _et al._ (2015); Intrieri and Gigli (2016); Bok  _et al._ (2018);
Sapsis (2018); Hanifi  _et al._ (2020); Morozov  _et al._ (2020); Scheffer
(2020); Rundle  _et al._ (2021); Bletery and Nocquet (2023); Cerini, Childs,
and Clements (2023); Domeisen  _et al._ (2023); Lehnertz, Bröhl, and von Wrede
(2023); Flores  _et al._ (2024); Hardebeck  _et al._ (2024). Typical examples
include epileptic seizures and cardiac arrest, population extinctions, market
crashes, mass panics, wars, earthquakes, tsunamis, extreme weather events,
rogue waves, or large-scale blackouts in power supply networks. Observations
and model simulations indicate that such critical transitions often occur when
changing conditions pass a critical or tipping point Horsthemke and Lefever
(1984); Kuehn (2011); Ashwin  _et al._ (2012); Feudel (2023). Both reliable
detection and forecasting of critical transitions are of utmost importance in
many scientific fields. If critical catastrophic transitions occur in an
unexpected way, they often do not allow for developing adaptation and/or
mitigation strategies. It is thus crucial to understand the mechanisms
underlying critical transitions, to identify early-warning indicators of such
transitions, and to reliably detect them in time series of suitable system
observables to enable forecasts.

The real-world systems referred to above can be considered as non-autonomous
dynamical systems Sell (1967); Kloeden and Rasmussen (2011), for which the
evolution of the state variable ${\bf x}\in\mathbb{R}^{d}$ is governed by

$\frac{\mathrm{d}{\bf x}(t)}{\mathrm{d}t}=f(t,{\bf
x}(t),\beta(t)),\qquad\frac{\partial f}{\partial t}\neq 0.$ (1)

Such systems are usually subject to time-dependent forcing as well as to
additive and/or multiplicative dynamical noise, which – together with the
various possible transition mechanisms (bifurcation-, noise- , rate-, or
shock-induced tipping Horsthemke and Lefever (1984); Kuehn (2011); Ashwin  _et
al._ (2012); Feudel (2023); Pavithran, Midhun, and Sujith (2023)) – poses
particular challenges for a time-series-analysis-based detection of critical
transitions.

The last years have seen a rapid increase of data- and theory-driven
approaches that aim at gaining deeper insights into non-autonomous dynamics
Clemson and Stefanovska (2014), into the mechanisms underlying critical
transitions and at their detection and forecasting. Among others, methods
based on nonlinear time-series analysis Kantz and Schreiber (2003), complex
network theory Boccaletti  _et al._ (2006); Zou  _et al._ (2019), or symbolic
time-series representation Amigó (2010) have been demonstrated to be useful
for specific systems, however, the general applicability of these methods
remains an issue to be investigated. Importantly, and with an eye on field
applications, it is crucial to estimate the reliability and significance of
indicators for critical transitions.

In this review, we summarize recent conceptual and methodological developments
that aim at a time-series-analysis-based investigation of critical transitions
in real-world non-autonomous systems. We critically assess the various steps
of investigation – from the experimental observation (Sect. II) via means to
cope with non-stationarity (Sect. III) and time-series analyses (Sect. IV) to
the offline detection of transitions (Sect. V) and (statistical) means to
evaluate their reliability (Sect. VI) – and highlight areas that are under
active investigation and that promise to provide new insights. Eventually, we
briefly discuss possibilities for time-series-analysis-based online detections
of critical transitions that could contribute to further increase reliability
of forecasts.

##  II Experimental observation of the dynamics of real-world non-autonomous
systems

For many real-world non-autonomous dynamical systems the governing equations
of motion are unknown, and quite often one lacks detailed knowledge about the
mechanisms that may underlie transitions. In such cases, one typically resorts
to the analysis of time series of suitable observables $\left\\{s(t)=s({\bf
x}(t)\right\\}$ with the measurement function
$s:\mathbb{R}^{d}\rightarrow\mathbb{R}$ and the discretized time $t=N\Delta t$
(with the number of data points $N$, and $\Delta t$ is the sampling interval).
This approach is, however, associated with some difficulties that one should
be aware of in order to avoid misinterpretations.

Frequently, one has access to a few (or even to only one) dynamical variables,
and one should consider in advance whether these are “good” observables
Kwasniok and Smith (2004); Mignan (2014); Mignan  _et al._ (2021).

Lacking detailed knowledge about the system under investigation, one is faced
with the problem of how to sufficiently sample the system dynamics in time and
space. This requires choosing a priori various recording parameters such as
number and placement of sensors, the sampling interval $\Delta t$, filter and
amplifier settings, and precision of analogue-to-digital conversion. The well-
known Nyquist–Shannon sampling theorem Nyquist (1928); Shannon (1949); Jerri
(1977) may have only limited significance, particularly if the full range of
dynamical behaviors and their temporal and spatial scales are not known a
priori. If the system contains nonlinearities, band-limiting possibly chaotic
dynamics using inappropriate filters may lead to severe misinterpretations
Kantz and Schreiber (2003). Lacking detailed knowledge about the system’s
spatial scales and about how – if at all – to decompose it into subsystems,
one typically resorts to a grid-like arrangement of sensors (Fig.1). In case
of complex coupling topologies between subsystems, such a placement can lead
to severe misinterpretations Bialonski, Horstmann, and Lehnertz (2010); Porz,
Kiel, and Lehnertz (2014).

Figure 1: Sampling the dynamics of a spatially extended system – for which the
actual division into subsystems (black dotted lines) is not known a priori –
is often performed with a grid-like arrangement of sensors (blue circles). An
associated problem is spatial oversampling whereby several sensors record the
dynamics of a given subsystem. Such redundant and mostly noisy information
might be hard to distinguish from synchronization or spreading phenomena.
Another problem is related to sensors that were happened to be placed such
that they capture the dynamics of neighboring subsystems. Disentangling
superimposed dynamics is hard without having the necessary a priori
information about the subsystems.

A widely used approach to study a system’s dynamics is probing, i.e.,
investigating the system’s response (relaxation dynamics) to small
perturbations. Such an approach, however, might not be feasible for any system
and more importantly, it involves a high level of risk of inducing a
transition to a new state with possibly undesirable properties – you may not
know in advance how big a small perturbation actually is. From the point-of-
view of time-series analysis, relaxation dynamics are typically reflected in
noisy, short-lasting transient signals that require dedicated denoising
techniques Effern  _et al._ (2000) or multiple realizations Jánosi and Tél
(1994); Dhamala, Lai, and Kostelich (2001); Andrzejak, Ledberg, and Deco
(2006); Łeski and Wójcik (2008); Komalapriya  _et al._ (2008); Wagner, Fell,
and Lehnertz (2010); Martini  _et al._ (2011); Ma  _et al._ (2014); Wang, Lai,
and Grebogi (2016) that may also be used to enhance time resolution by
replacing the temporal average with an ensemble average. The latter, however,
implicitly assumes stationarity of the system (at least on the considered
timescale) and comparability of relaxation dynamics and of transitions.

An alternative to probing is observing a system over timescales long enough to
capture all relevant aspects of its dynamics (e.g. “normal” dynamics, critical
and non-critical transitions). For real-world non-autonomous dynamical
systems, however, this approach requires dedicated methods to cope with the
inherently nonstationary dynamics (see Sect. III), with possible very long
transients Kaneko (1990); Tél and Lai (2008); Morozov  _et al._ (2020); Meyer-
Ortmanns (2023), or even with non-convergent (non-asymptotic) behaviors
Ansmann, Lehnertz, and Feudel (2016); Nandan and Koseska (2023). It should be
determined in advance whether continuous data collection is required or
whether collecting snippets of data suffices for an adequate characterization
of the system’s dynamics. Likewise, it is advisable to keep the aforementioned
recording parameters constant throughout the observation time as these could
easily represent time-dependent confounders.

##  III Coping with non-stationarity

Non-autonomous dynamical systems are non-stationary, non-ergodic systems that
do not obey Birkhoff’s ergodic theorem equaling time and ensemble averages
Boltzmann (1877); Birkhoff (1931). This property is usually regarded as an
obstacle as it poses special challenges for time-series-analysis methods.

A (stochastic) process is called strictly (or strongly) stationary if the
distribution of its states over an ensemble of realizations of that process
does not depend on time Priestley (1988). This implies constancy of all
statistical moments and all joint statistical moments. Within the scope of
field applications, stationarity of a system can only be evaluated with
respect to the observation time. The vast majority of time-series-analysis
methods (see Sect. IV) require the system under investigation to be at least
approximately stationary (constancy of first- and second-order statistical
moments) in order to allow for a robust and reliable characterization.

Figure 2: A frequently used approach to investigate the dynamics of a non-
stationary system is to estimate a characterizing metric $m$ (cf. Sect. IV)
using the data of successive segments (or windows; marked in gray) of time
series of system observables. During each segment, the system is assumed to be
(approximately) stationary (moving-window analysis). The resulting time-
dependent sequence $m(t_{\rm w})$, where $t_{\rm w}$ denotes the sequence’s
temporal resolution, is then used to detect transitions as well as to identify
indicators for the latter.

The most common way of dealing with a non-stationary system is to cut the time
series into successive (non-)overlapping segments (or windows Prabhu (2014))
during which the system’s dynamics can be regarded as approximately stationary
(sliding or rolling or moving-window analysis) Kitagawa (1987); Dahlhaus
(1997); Huang  _et al._ (1998); Verdes  _et al._ (2001); Kantelhardt  _et al._
(2002); Fukuda, Stanley, and Amaral (2004); Wu  _et al._ (2007); Clemson and
Stefanovska (2014); Lehnertz  _et al._ (2017); Podobnik and Stanley (2008);
Rhif  _et al._ (2019). A characterizing metric is then calculated for each
data window by employing some time-series-analysis method (see Fig. 2 and
Sect. IV).

Instead of identifying (approximately) stationary data segments, one might
also consider non-stationarity as an interesting aspect of the dynamics.
Methods that allow tracing non-stationarity are based e.g. on the concept of
overembedding Hegger  _et al._ (2000); Verdes, Granitto, and Ceccatto (2006);
De Domenico and Latora (2010) that treats time-varying control parameters as
independent dynamical variables. Other methods make use of recurrence
quantification analysis Manuca and Savit (1996); Kennel (1997); Gao (2001);
Rieke  _et al._ (2002, 2004); Facchini, Kantz, and Tiezzi (2005); Chen and
Yang (2012), of state-space-based cross-predictions Schreiber (1997), or of
symbolic data compression Kennel and Mees (2000), to name just a few.

Regardless of the perspective taken, all approaches require some statistical
test Priestley (1988); Robinson (1994); Witt, Kurths, and Pikovsky (1998) to
assess the quality of differentiation/characterization.

##  IV Time-series-analysis

Time-series analysis comprises methods for analyzing a single or multiple
sequences of observations collected over a certain period of time in order to
extract meaningful statistics and other characteristics of the data. A
plethora of linear and nonlinear time-series-analysis methods is available
that can be used to detect and characterize – in a data-driven way – different
aspects and properties of a system’s dynamics Champeney (1973); Brody  _et
al._ (1981); Haykin (1983); Grassberger, Schreiber, and Schaffrath (1991);
Abarbanel  _et al._ (1993); Honerkamp (1993); Kaplan and Glass (1995);
Percival and Walden (2000); Pikovsky, Rosenblum, and Kurths (2001); Daw,
Finney, and Tracy (2003); Kantz and Schreiber (2003); Reinsel (2003); Keller
and Sinn (2005); Lütkepohl (2005); Hlaváčková-Schindler  _et al._ (2007);
Marwan  _et al._ (2007); Lacasa  _et al._ (2008); Anderson (2011); Friedrich
_et al._ (2011); Von Toussaint (2011); Bradley and Kantz (2015); Webber, Jr.
and Marwan (2015); Stankovski  _et al._ (2017); Tabar (2019); Zou  _et al._
(2019); Hamilton (2020); Edinburgh, Eglen, and Ercole (2021); Datseris and
Parlitz (2022); Fokianos  _et al._ (2022); Nikakhtar  _et al._ (2023); Tabar
_et al._ (2024). Some methods may also be used for forecasting (or
anticipation) purposes.

Analysis methods can be classified into two main groups.

With univariate methods, the above-mentioned applications are limited to a
single time series. Exemplary applications include estimating statistical
moments (e.g., mean, standard deviation, skewness, kurtosis), characteristics
of the auto-correlation function (e.g., decay time) or the power spectrum
(e.g., spectral power in some frequency band), Kramers-Moyal coefficients
(e.g., drift and diffusion), or state-space-based quantities such as
dimensions, entropies, stability indicators like Lyapunov exponents, or
characteristics derived from recurrence quantification analysis. The latter
quantities require an appropriate reconstruction of the dynamics in state-
space from a single (or only a few) time series Packard  _et al._ (1980);
Takens (1981); Sauer, Yorke, and Casdagli (1991); Casdagli  _et al._ (1991);
Kennel, Brown, and Abarbanel (1992); Kugiumtzis (1996); Cao (1997); Cellucci,
Albano, and Rapp (2003); Vlachos and Kugiumtzis (2010); Krämer  _et al._
(2021).

With bivariate (or, in general, multivariate) methods, investigations mostly
concentrate on characterizing interdependences between pairs (or, in general,
n-tuples) of time series. Exemplary applications include estimating
characteristics of the cross-correlation function (e.g., interaction delay) or
the cross-spectrum (e.g., coherence function), properties of interactions
(strength, direction, functional form) and of synchronization phenomena, or
characteristics derived from network theory.

Since there is no one-fits-all time-series-analysis method for all types of
data, the choice of a specific method depends on the system under
investigation and on the research question being asked. In addition, since one
is interested in the physical mechanism(s) underlying the observed dynamics
(or properties thereof), the choice of the method needs to be justified by a
hypothesis about the appropriate data model. Given that almost all time-series
analysis methods reduce the information content of some time series to some
characteristic number or metric, interpretability of the latter is only
possible if it has a specific meaning within the model framework. Only in that
case does the reduced information increase our knowledge about the system
under investigation. If the data to be analyzed does not pertain to the
appropriate model class, the chosen metric may not make much sense, even if
its numerical value can be calculated for the given time series using a
numerical algorithm.

For the majority of time-series-analysis methods, a reduction of the
information content of some time series involves time-, space-, or ensemble-
averages or combinations thereof in order to achieve a sufficient accuracy and
robustness against noise contaminations. It is often assumed that the more
data enters some algorithm the higher is the method’s accuracy and thus the
reliability of the chosen metric. However, many time-series-analysis methods,
or rather the concepts/theories from which they originate, (implicitly) assume
the system under investigation to be stationary. As already mentioned above,
investigations of non-autonomous dynamical systems often rely on a moving-
window analysis to account for non-stationarity. A shorter analysis window is
usually associated with approximate stationarity, but one needs to take into
account that the decreased amount of data entering some analysis method may
decrease the reliability of a characterizing metric. Thus, the choice of a
window length $N_{\rm{w}}\ll N$ is often a compromise between the required
statistical accuracy for the estimation of a metric and approximate
stationarity within a window length.

Calculating a metric using a time-series-analysis method is usually
accompanied by the choice of algorithmic parameters. Depending on the research
question, algorithmic parameters are often tuned to achieve an optimum
characterization of the system’s dynamics. For an investigation of long-term
observations and using a moving-window analysis, however, tuning parameters
for each window might not be advisable. The resulting (time-dependent)
sequence of a metric’s numerical values could then simply reflect this tuning-
per-window which can lead to severe misinterpretations. It might thus be
advisable to fix all necessary pre-processing steps and keep a priori best-
chosen algorithmic parameters constant.

##  V Detecting transitions

A number of metrics have been proposed in recent years to detect critical
transitions based on bifurcations and the associated phenomenon of critical
slowing down, i.e., an increasingly slow recovery from small perturbations
Dakos  _et al._ (2008); Scheffer  _et al._ (2009, 2012); Dai  _et al._ (2012);
Lenton  _et al._ (2012). With approaching the bifurcation (tipping point), the
time needed to recover from perturbations becomes longer and hence the system
dynamics becomes more correlated with its past, leading to an increase in the
lag-1 autocorrelation and to spectral reddening Bury, Bauch, and Anand (2020).
Since perturbations accumulate, one observes an increase in the size of the
fluctuations Kubo (1966) as assessed with variance or other higher-order
statistical moments. These classic statistical indicators have been used to
investigate critical slowing in various contexts Cotilla-Sanchez, Hines, and
Danforth (2012); Dakos  _et al._ (2012); van de Leemput  _et al._ (2014);
Meisel  _et al._ (2015); Perry, Livina, and Niewczas (2015); Nazarimehr  _et
al._ (2020); Brookes  _et al._ (2021); George, Kachhara, and Ambika (2021);
Southall  _et al._ (2021); Dablander  _et al._ (2022); Deb  _et al._ (2022);
Ditlevsen and Ditlevsen (2023); Mathevet  _et al._ (2024) and were claimed to
be generic. They continue, however, to be critically discussed from various
perspectives Ditlevsen and Johnsen (2010); Boettiger and Hastings (2012a,
2013); Boettiger, Ross, and Hastings (2013); Kéfi  _et al._ (2013); Guttal,
Jayaprakash, and Tabbaa (2013); Dakos  _et al._ (2015); Dai, Korolev, and Gore
(2015); Diks, Hommes, and Wang (2015); Wagner and Eisenman (2015); Zhang,
Kuehn, and Hallerberg (2015); Guttal  _et al._ (2016); Gsell  _et al._ (2016);
Milanowski and Suffczynski (2016); Dutta, Sharma, and Abbott (2018); Wen,
Ciamarra, and Cheong (2018); Romano and Kuehn (2018); Arumugam  _et al._
(2019); Clements, McCarthy, and Blanchard (2019); Gatfaoui and De Peretti
(2019); Jäger and Füllsack (2019); Wilkat, Rings, and Lehnertz (2019); Marconi
_et al._ (2020); van der Bolt, van Nes, and Scheffer (2021); Lapeyrolerie and
Boettiger (2023); O’Brien  _et al._ (2023); Proverbio, Skupin, and Gonçalves
(2023).

The quest for more robust and possibly more widely applicable indicators that
may be sensitive to critical transitions other than those related to
bifurcations – such as boundary crisis Grebogi, Ott, and Yorke (1983); Osinga
and Feudel (2000), blowout bifurcations Ott and Summerer (1994); Zhang  _et
al._ (2020), saddle escape Kuehn, Zschaler, and Gross (2015), oscillation
death Koseska, Volkov, and Kurths (2013); Zou  _et al._ (2021), or explosive
synchronization Boccaletti  _et al._ (2016); Kuehn and Bick (2021) – has led
to the development of metrics that are based on other time-series-analysis
concepts. Among others, these include synchronization Mormann  _et al._ (2003,
2005); Winterhalder  _et al._ (2006); Kuhlmann  _et al._ (2010), symbolic
dynamics Ray (2004); Chin, Ray, and Rajagopalan (2005); Lehnertz and Dickten
(2015), complex networks Liu  _et al._ (2015); Peng  _et al._ (2019); Rings
_et al._ (2019); Rings, von Wrede, and Lehnertz (2019); Ludescher  _et al._
(2021); Lehnertz, Bröhl, and von Wrede (2023), algebraic topology Mittal and
Gupta (2017); Syed Musa  _et al._ (2021); Ghil and Sciamarella (2023),
autoregressive modeling Giacomini and Rossi (2009); Ghalati  _et al._ (2019),
entropy concepts Liu  _et al._ (2012); Meng  _et al._ (2020); Pavithran, Unni,
and Sujith (2021); Tirabassi and Masoller (2023); Deb and Dutta (2024),
reconstruction of Jacobian matrices Barter  _et al._ (2021), (nonstationary)
dynamical modeling Franzke (2013); Kwasniok (2013, 2015, 2018); Din, Liang,
and Zhou (2018), or the Kramers-Moyal expansion Lehnertz, Zabawa, and Tabar
(2018); Tabar (2019); Arani  _et al._ (2021); Heßler and Kamps (2022, 2023).

All these time-series-analysis techniques have in common that, when
investigating real-world non-autonomous dynamical systems, moving-window
analyses of some system observable(s) yield time-dependent sequence(s) of a
characterizing metric
$\left\\{m(t_{\rm{w}}),m(2t_{\rm{w}}),m(3t_{\rm{w}}),\dots\right\\}$. These
sequences have a temporal resolution $t_{\rm{w}}$ that is given by the window
length $N_{\rm{w}}$ and can subsequently be used to detect transitions as well
as to identify indicators for the latter.

There are two major techniques in use to detect transitions, namely threshold
crossing and change point detection (see Fig.3). The latter is sometime also
referred to as anomaly detection. Both techniques are often employed as
offline methods that retrospectively detect changes when the data has already
been gathered and using information on transitions that have taken place in
the past. When employed as online methods, they aim to detect changes as soon
as they occur in a real-time setting. This may be achieved with machine
learning techniques Lim  _et al._ (2020); Kong  _et al._ (2021); Ray,
Chakraborty, and Ghosh (2021); Patel  _et al._ (2021); Patel and Ott (2023);
Köglmayr and Räth (2024) or Bayesian inference Shcherbakov  _et al._ (2019);
Heßler and Kamps (2022).

Figure 3: Illustration of techniques to detect transitions. With threshold
crossing (top), the onset of a transition is related to the first time point
$t_{\rm{crit}}$ when a characterizing metric $m$ crosses a pre-defined
threshold $\Theta$ (either from above or from below). With change point
detection (bottom), specific points in time (blue dots) are identified where
the sequence of metric $m$ exhibits a significant change either in its
deterministic content or underlying stochastic distribution. The onset of a
transition can be related, for example, to the first identified change point.

### Threshold crossing

The onset of a transition is given by the first timepoint $t_{\rm{crit}}$ when
the metric $m$ exceeds or falls below some pre-defined threshold $\Theta$
Lehnertz and Elger (1998); Streeter and Dugmore (2013); Bialonski, Ansmann,
and Kantz (2015); Zhang, Kuehn, and Hallerberg (2015); Grziwotz  _et al._
(2023). The latter may be defined as the mean plus or minus two standard
deviations estimated from preceding data (two-sigma rule Benjamin  _et al._
(2018)). Note, however, that in driven non-stationary systems statistical
moments of a sequence of some metric may undergo fluctuations on timescales
much longer than the one for which the system’s dynamics has been regarded as
approximately stationary Lehnertz, Rings, and Bröhl (2021) (see Sect. III). In
such a case it might be advisable to replace the fixed threshold by an
adaptive one Dahlhaus and Neumann (2001). Since the rather simple threshold-
crossing-based criterion might be prone to isolated outliers, one can also
consider the onset of a transition as the first timepoint for which multiple
consecutive data points
$\left\\{m(t_{\rm{crit}}),m(t_{\rm{crit}}+t_{\rm{w}}),m(t_{\rm{crit}}+2t_{\rm{w}}),\dots\right\\}$
fulfill the threshold criterion. Indeed, one can expect the number of
consecutive data points fulfilling the threshold criterion to increase as the
system approaches a critical transition. If assumptions can be made about how
a metric changes during a critical transition (e.g. linear increase or
decrease), one may eventually establish the presence of trends (e.g., using
linear regression) in order to further substantiate the evidence of a critical
transition.

### Change point detection

This technique aims to identify specific points in time where the sequence of
metric $m$ exhibits a significant change either in its deterministic content
or underlying stochastic distribution Carlstein (1988); Basseville, Nikiforov
_et al._ (1993); Aminikhanghahi and Cook (2017); Cabrieto  _et al._ (2017,
2018); Truong, Oudre, and Vayatis (2020); Bagniewski and Ghil (2021); Bury
_et al._ (2021); Gilarranz  _et al._ (2022). In general, change point
detection concerns both detecting whether or not a change has occurred, or
whether several changes might have occurred, and identifying the times of any
such changes. Depending on the application, this may involve measuring the
statistical (dis-)similarity between two windows of data to either side of a
purported change point or identifying changes in the mean, variance,
autocorrelation, or spectral density of the sequence Truong, Oudre, and
Vayatis (2020); De Ryck, De Vos, and Bertrand (2021). Note that the latter
also requires the choice of a suitable and a priori defined window. In
addition, the same restrictions apply in the case of driven non-stationary
systems as with threshold crossing.

##  VI Testing the reliability of offline detections

Consider an offline time-series-analysis-based detection of critical
transitions in some real-world non-autonomous dynamical system for which we
know the exact onset times of critical transitions. The applied analyses
techniques are assumed to be sensitive enough to detect and characterize
critical transitions, based on prior investigations (e.g., with toy models).
We observe additional changes in the temporal sequence of the employed metric
that would qualify as indicative of critical transitions, but we know a
posteriori that the system has solely undergone transitions to some non-
critical states – we refer to these as non-critical transitions. Such a
situation can occur, for example, when analyzing brain dynamics and employing
time-series-analysis techniques that can not (or only insufficiently)
distinguish between a transition to a pre-seizure state (critical transition)
and transitions to behavioral states like sleep or wakefulness. We consider
all other changes as false detections.

Figure 4: Top: exemplary outcome of an offline detection of the onsets of
critical transitions (CT, red). Onsets of non-critical transitions (nCT) are
marked black. Detections are labeled with the letter “d”. Bottom: the duration
of a critical ($\Delta_{\rm CT}$, red) and a non-critical ($\Delta_{\rm nCT}$,
black) transition can be defined as differences between the respective ending
and onset times. Onset times are denoted by $t_{\rm o}$; for the critical
transition, the ending of the transitory phase coincides with the occurrence
of the extreme event at $t_{\rm e}$ (note that for $\Delta_{\rm CT}\to 0$, the
transition would be classified as abrupt and there are no early warning
signs); for the non-critical transition, the ending of the transitory phase
coincides with the return of the system to its normal dynamics at $t_{\rm r}$.

For the (quite typical) detection example shown in the upper part of Figure 4,
the following interpretations can be made:

  * •

since the method detected only half of the critical transitions, it appears to
be not very sensitive;

  * •

given that the method detected two out of three non-critical transitions, it
might be more sensitive to the latter type of transitions;

  * •

there are more false than true detections, so the method appears to be not
very specific;

  * •

there are two detections that precede the onsets of the critical transitions,
however, since one detection coincides with the onset of a non-critical
transition the method appears to have only a limited predictive significance;

  * •

since there are three detections that follow the offsets of critical and non-
critical transition, one might speculate about the method’s capability to
detect the offset of such transitions.

This example clearly points to the need to check reliability and quality of
offline time-series-analysis-based methods for the detection of critical
transitionsBoettiger and Hastings (2012b). For this purpose, various
statistical tests can (and should) be applied in order to avoid drawing false
conclusions and to avoid wishful thinking.

Tests range from simple to advanced and require knowledge about the occurrence
times of (non-)critical transitions (or events) as well as about
characteristics of the respective transitory phases that may differ from event
to event (in order to simplify things, one could treat false detections as
non-critical transitions). Depending on the application, these characteristics
include e.g. the durations of transitory phases ($\Delta_{\rm CT}$ and
$\Delta_{\rm nCT}$; see lower part of Figure 4), the maximum deflection from
the threshold, or characteristics of a trend function Bunde, Ludescher, and
Schellnhuber (2022).

### Distinguishability of non-critical and critical transitory phases

A simple though not very reliable way to assess detection performance of
offline methods is to use statistical approaches Daniel (2000) to verify
distinguishability of non-critical and critical transitory phases by comparing
the distributions – or moments thereof – of their characteristics (durations,
maximum deflections, etc.). The equality of distributions can be tested
nonparametrically Kolmogorov (1933) or rank-based Mann and Whitney (1947). For
Gaussian-distributed data, equality of means and standard deviations can be
evaluated with the Student’s t-test and the F-test, respectively. For non-
Gaussian-distributed data, these are the median test or the Mann-Whitney U
test and the Conover squared ranks test. It should be noted, however, that the
tests assume that the data were taken independently (from some distribution).
This implies no memory, no dynamics, time is not important, and stationarity
of the investigated system.

### Sensitivity analysis

Another simple way to assess detection performance of offline methods is to
count the number of correct detections of critical transitions. This approach
allows drawing some conclusions about a method’s sensitivity but it does say
anything about a method’s specificity. Sensitivity refers to the probability
of a positive test result (“is critical”), conditioned on each detection truly
being positive. Specificity refers to the probability of a negative test
result (“is non-critical”), conditioned on each detection truly being
negative. Also, this approach does not take into account false detections (see
below), and one quite often finds studies that report on sensitivity of a
detection method only. Note that a detection method can easily be tuned
through in-sample optimization to achieve 100 % sensitivity when ignoring its
specificity. Lacking relevant information and, more importantly, considering
the fact that critical transitions are typically rare events, this simple way
is not very robust and largely unsuited for analyses of empirical data.

### ROC analysis

A more robust way to assess detection performance is the Receiver Operating
Characteristics (ROC) analysis Yerushalmy (1947); Fawcett (2006). In addition
to counting the number of correct (true positive) and missing (true negative)
detections of critical transitions as estimates for sensitivity and
specificity, this approach also takes into account false detections. A false
positive detection refers to a non-critical transition that was erroneously
detected as critical transition (type I error). A false negative detection
refers to a critical transition that was erroneously detected as non-critical
transition (type II error). One then estimates the method’s sensitivity (or
the true positive rate; TPR) as well as $1-$specificity (or the false positive
rate; FPR) at each setting of the threshold value $\Theta$ to obtain the so
called ROC curve Egan (1975) (see Fig. 5). In this ROC space, the best
possible detection method would yield a point in the upper left corner (or
lower right depending on interpretation and change of metric $m$),
representing 100 % sensitivity (no false negatives) and 100 % specificity (no
false positives). In contrast, a detection method working at random would give
a point along the diagonal from the bottom left to the top right corners
(regardless of the positive and negative base rates; cf. random guessing).

Figure 5: Performance of a detection method can be evaluated on the basis of a
receiver operating characteristic (ROC) curve that relates true positive rates
to false positive rates. The area under the curve (AUC) can be used to
quantify the detection method’s performance (an ideal detection method has an
AUC of 1).

An often used reduction of the ROC Curve to a single number is calculating the
so called area under the ROC curve (AUC). Depending on application, AUC can be
defined as the value of the area under the whole curve which yields 1.0 for
the best possible detection method and 0.5 for a random detection Mormann  _et
al._ (2005). Alternatively, AUC can be defined as the (absolute) value of the
area between the ROC curve and the diagonal which yields values between 0.0
(random detection) and 0.5 (best detection).

High AUC values for various detection methods continue to be reported from
different disciplines Drake (2013); Zhang, Kuehn, and Hallerberg (2015);
Brinkmann  _et al._ (2016); Cavaliere  _et al._ (2016); Kuhlmann  _et al._
(2018a); Yang  _et al._ (2018); Gatfaoui and De Peretti (2019); Brett  _et
al._ (2020); Bury  _et al._ (2021); Southall  _et al._ (2021); Tredennick  _et
al._ (2022); Dylewsky  _et al._ (2023); Gómez-Nava  _et al._ (2023). Such
values, however, can give a false impression about a method’s genuine
detection reliability, particularly for imbalanced probability distributions
for true and false positive detections, which is the case for critical
transitions that are typically rare events. This imbalance can lead to a bias
towards a predominant number of detections of non-critical transitions and can
render an interpretation of AUC values oversimplistic and even misleading
Weiss (2004); Mormann  _et al._ (2005); Ben Bouallègue and Richardson (2022);
West  _et al._ (2023). Such misinterpretations can be avoided with the help of
surrogate-assisted ROC analyses that were introduced in the field of
prediction of epileptic seizures Mormann  _et al._ (2007); Kuhlmann  _et al._
(2018b) in the context of detection of critical transitions (here: detection
of a pre-seizure state).

### Surrogate-guided ROC analysis

A simple and straightforward ansatz consists of comparing a method’s detection
performance against the one of a random detector Winterhalder  _et al._
(2003); Schelter  _et al._ (2006); Feldwisch-Drentrup  _et al._ (2011); Mader
_et al._ (2014). To do so, artificial detections are generated at random times
(e.g., Poisson-distributed) but using the same rate as that of the method
under investigation. This easy and computationally fast approach might require
adjustments in case of (near-)periodic occurrences of transitions and it might
not be generally applicable. A more advanced though more time-intensive ansatz
makes use of null-hypothesis-based resampling or bootstrapping methods Efron
(1982); Efron and Tibshirani (1998), also known as surrogate techniques in
time series Schreiber and Schmitz (2000); Lucio, Valdés, and Rodríguez (2012);
Lancaster  _et al._ (2018) and network analysis Ansmann and Lehnertz (2011,
2012); Bialonski (2012); Laut and Räth (2016); Wiedermann  _et al._ (2016);
Stahn and Lehnertz (2017); Chorozoglou and Kugiumtzis (2019). One such
technique Andrzejak  _et al._ (2003) consists of a random shuffling of the
time intervals between successive (true) detections of critical transitions to
generate artificial detection timepoints. A related technique Kreuz  _et al._
(2004) leaves the (true) detection timepoints unchanged and instead performs a
constrained randomization of the temporal sequence of a characterizing metric
using e.g. simulated annealing Bertsimas and Tsitsiklis (1993). Both these
techniques require formulating an appropriate null-hypothesis as well as a
robust (non-parametric) test statistic to assess the statistical significance
of the results obtained. The number of independent surrogates $N_{\rm surr}$
determines the nominal size of the test statistic (i.e., the probability
$\alpha$ of making a type I error; $1/N_{\rm surr}\leq\alpha$). The
performance (as assessed with AUC) of the detection method under investigation
is then compared with the performances of a surrogate ensemble. If the
method’s performance exceeds to performances of the surrogates, then the
method can be considered – with greater confidence – as performing better than
chance. Reporting the “number of sigmas” can be misleading since it is
accompanied by strict assumptions about the underlying distributions of
performance values. Surrogate-based approaches can account for a possibly non-
random occurrence of critical transitions which might be encountered with
driven non-stationary systems.

### Time series surrogates

Null-hypothesis-based surrogate techniques are also being used to generate
ensembles of time series that share relevant statistical properties with the
original time series of some suitable observable except for properties that
are assumed to be indicative of a critical transition. As an example, we
mention trends of indicators for critical slowing down such as variance and
lag-1 autocorrelation. Techniques to generate such surrogate time series
include random shuffling, autoregressive model-based methods, or iterative
amplitude-adjusted Fourier transform Schreiber and Schmitz (2000) and are
often used to evaluate the significance of the aforementioned indicators Dakos
_et al._ (2012); Kramer  _et al._ (2012); George, Kachhara, and Ambika (2021);
Boettner and Boers (2022); Chen, Ghadami, and Epureanu (2022); Pal, Deb, and
Dutta (2022); Bochow and Boers (2023). Techniques require formulating an
appropriate null-hypothesis as well as a robust (non-parametric) test
statistic to assess the statistical significance of the results obtained. The
number of independent surrogate time series determines the nominal size of the
test statistic.

Before closing this section, it is important to recall the following caveats
when interpreting an offline detection method’s reliability evaluated with the
aforementioned testsShmueli (2010):

  * •

statistical tests can neither prove the correctness of an offline detection
method’s reliability nor can they prove the existence of a critical
transition;

  * •

a rejection of a null hypothesis only provides a necessary but not a
sufficient condition for a method to indicate reliable offline detection of a
critical transition;

  * •

an acceptance of a null hypothesis does not indicate its correctness;

  * •

whenever a null hypothesis is rejected, it is always very important to keep in
mind that the complementary hypothesis is typically very comprehensive and
might include many different reasons that are possibly responsible for this
rejection;

  * •

consider including other (statistical) properties of your data and detection
method into the null hypothesis.

##  VII Prospects for time-series-analysis-based online detections of critical
transitions

So far, only a few studies reported on above-chance-level online detections of
critical transitions in real-world non-autonomous systems from empirical data
Cook  _et al._ (2013); Ludescher, Bunde, and Schellnhuber (2023). Findings
suggest that it is possible to forecast the occurrence of a recurrent extreme
event in some cases, however, predicting exact timing and magnitude of an
event is much more difficult. Forecasts should thus be probabilistic Gneiting
and Katzfuss (2014) instead of aiming at fully deterministic predictions.

When judging the reliability of probabilistic forecasts (or forecast skill) of
rare and recurrent extreme events, the statistical tests discussed in Sect. VI
can be oversimplistic or even misleading Tashman (2000); Ben Bouallègue and
Richardson (2022). Proper scoring rules and strictly proper scores Gneiting,
Balabdaoui, and Raftery (2007); Gneiting (2011); Lerch  _et al._ (2017);
Gneiting  _et al._ (2023) and uncertainty quantification Berger and Smith
(2019) appear to be more suitable, however, their use for time-series-
analysis-based online detections of critical transitions requires further
conceptual and methodological developments.

##  VIII Conclusions

The time-series-analysis-based detection of critical transitions in real-world
non-autonomous systems is a young area of research that carries the potential
to advance understanding and reliable forecasting of such nonlinear behavior.
In this paper, I provided an overview of and critically assessed the various
involved steps of investigation: from the data recording via coping with non-
stationarity when analyzing time series of system observables to evaluating
the reliability of offline and online detections. I highlighted pros and cons
to stimulate further developments that require a close interlinking of theory
and experiment.

Time-series analysis methods in use appear to be suitable to allow detection
and sufficient characterization of bifurcation-induced critical transitions in
low-dimensional systems. Nevertheless, their suitability for these types of
transition in high-dimensional systems as well as for other transition
mechanisms remains to be shown. Is is conceivable that the latter require
novel developments also in terms of robustness, appropriateness for non-
stationary systems, and – ideally – general interpretability. Likewise, the
same applies to methods to test the reliability of detections.

Extreme events occur in a variety of natural, technical, and societal systems
and their frequency appears to be increasing. Their catastrophic consequences
and their low-probability, high-impact nature not only calls for improving our
understanding of generating mechanisms but also to provide reliable early
warnings to enable the development of adaptation and/or mitigation strategies.

###### Acknowledgements.

The author acknowledges fruitful discussions with Manuel Adams, Timo Bröhl,
Ulrike Feudel, and Thorsten Rings.

## AUTHOR DECLARATIONS

### Conflict of Interest

The author has no conflicts to disclose.

### Data availability

Data sharing is not applicable to this article as no new data were created or
analyzed in this study.

## References

  * Albeverio, Jentsch, and Kantz (2006) S. Albeverio, V. Jentsch, and H. Kantz, eds.,  _Extreme events in nature and society_ , The Frontiers Collection (Springer, Berlin, 2006).
  * Basher (2006) R. Basher, “Global early warning systems for natural hazards: systematic and people-centred,” Phil. Trans. Roy. Soc. A: mathematical, physical and engineering sciences 364, 2167–2182 (2006).
  * Willoughby, Rappaport, and Marks (2007) H. E. Willoughby, E. Rappaport, and F. Marks, “Hurricane forecasting: The state of the art,” Nat. Hazards Rev. 8, 45–49 (2007).
  * Lenton (2011) T. M. Lenton, “Early warning of climate tipping points,” Nat. Clim. Change 1, 201 (2011).
  * Ansmann  _et al._ (2013) G. Ansmann, R. Karnatak, K. Lehnertz, and U. Feudel, “Extreme events in excitable systems and mechanisms of their generation,” Phys. Rev. E 88, 052911 (2013).
  * Meron (2015) E. Meron,  _Nonlinear physics of ecosystems_ (CRC Press, Boca Raton, FL, USA, 2015).
  * Trefois  _et al._ (2015) C. Trefois, P. M. Antony, J. Goncalves, A. Skupin, and R. Balling, “Critical transitions in chronic disease: transferring concepts from ecology to systems medicine,” Curr. Opin. Biotechnol. 34, 48–55 (2015).
  * Intrieri and Gigli (2016) E. Intrieri and G. Gigli, “Landslide forecasting and factors influencing predictability,” Nat. Hazards Earth Syst. Sci. 16, 2501–2510 (2016).
  * Bok  _et al._ (2018) B. Bok, D. Caratelli, D. Giannone, A. M. Sbordone, and A. Tambalotti, “Macroeconomic nowcasting and forecasting with big data,” Annu. Rev. Econ. 10, 615–643 (2018).
  * Sapsis (2018) T. P. Sapsis, “New perspectives for the prediction and statistical quantification of extreme events in high-dimensional dynamical systems,” Phil. Trans. Roy. Soc. A: Mathematical, Physical and Engineering Sciences 376, 20170133 (2018).
  * Hanifi  _et al._ (2020) S. Hanifi, X. Liu, Z. Lin, and S. Lotfian, “A critical review of wind power forecasting methods—past, present and future,” Energies 13, 3764 (2020).
  * Morozov  _et al._ (2020) A. Morozov, K. Abbott, K. Cuddington, T. Francis, G. Gellner, A. Hastings, Y.-C. Lai, S. Petrovskii, K. Scranton, and M. L. Zeeman, “Long transients in ecology: Theory and applications,” Phys. Life Rev. 32, 1–40 (2020).
  * Scheffer (2020) M. Scheffer,  _Critical transitions in nature and society_ , Vol. 16 (Princeton University Press, Princeton, New Jersey, 2020).
  * Rundle  _et al._ (2021) J. B. Rundle, S. Stein, A. Donnellan, D. L. Turcotte, W. Klein, and C. Saylor, “The complex dynamics of earthquake fault systems: New approaches to forecasting and nowcasting of earthquakes,” Rep. Prog. Phys.. 84, 076801 (2021).
  * Bletery and Nocquet (2023) Q. Bletery and J.-M. Nocquet, “The precursory phase of large earthquakes,” Science 381, 297–301 (2023).
  * Cerini, Childs, and Clements (2023) F. Cerini, D. Z. Childs, and C. F. Clements, “A predictive timeline of wildlife population collapse,” Nat. Ecol. Evol. 7, 320–331 (2023).
  * Domeisen  _et al._ (2023) D. I. Domeisen, E. A. Eltahir, E. M. Fischer, R. Knutti, S. E. Perkins-Kirkpatrick, C. Schär, S. I. Seneviratne, A. Weisheimer, and H. Wernli, “Prediction and projection of heatwaves,” Nat. Rev. Earth Environ. 4, 36–50 (2023).
  * Lehnertz, Bröhl, and von Wrede (2023) K. Lehnertz, T. Bröhl, and R. von Wrede, “Epileptic-network-based prediction and control of seizures in humans,” Neurobiol. Dis. 181, 106098 (2023).
  * Flores  _et al._ (2024) B. M. Flores, E. Montoya, B. Sakschewski, N. Nascimento, A. Staal, R. A. Betts, C. Levis, D. M. Lapola, A. Esquível-Muelbert, C. Jakovac, _et al._ , “Critical transitions in the Amazon forest system,” Nature 626, 555–564 (2024).
  * Hardebeck  _et al._ (2024) J. L. Hardebeck, A. L. Llenos, A. J. Michael, M. T. Page, M. Schneider, and N. J. van der Elst, “Aftershock forecasting,” Annu. Rev. Earth Planet. Sci. 52, 2.1–2.24 (2024).
  * Horsthemke and Lefever (1984) W. Horsthemke and R. Lefever,  _Noise-Induced Transitions. Theory and Applications in Physics, Chemistry and Biology_ (Springer, Berlin, 1984).
  * Kuehn (2011) C. Kuehn, “A mathematical framework for critical transitions: Bifurcations, fast-slow systems and stochastic dynamics,” Physica D 240, 1020–1035 (2011).
  * Ashwin  _et al._ (2012) P. Ashwin, S. Wieczorek, R. Vitolo, and P. Cox, “Tipping points in open systems: bifurcation, noise-induced and rate-dependent examples in the climate system,” Phil. Trans. Roy. Soc. A 370, 1166–1184 (2012).
  * Feudel (2023) U. Feudel, “Rate-induced tipping in ecosystems and climate: the role of unstable states, basin boundaries and transient dynamics,” Nonlin. Processes Geophys. Discussions 2023, 1–29 (2023).
  * Sell (1967) G. R. Sell, “Nonautonomous differential equations and topological dynamics. I. The basic theory,” Trans. Amer. Math. Soc. 127, 241–262 (1967).
  * Kloeden and Rasmussen (2011) P. E. Kloeden and M. Rasmussen,  _Nonautonomous dynamical systems_ (American Mathematical Soc., Providence, RI, USA, 2011).
  * Pavithran, Midhun, and Sujith (2023) I. Pavithran, P. Midhun, and R. Sujith, “Tipping in complex systems under fast variations of parameters,” Chaos: An Interdisciplinary Journal of Nonlinear Science 33, 081105 (2023).
  * Clemson and Stefanovska (2014) P. T. Clemson and A. Stefanovska, “Discerning non-autonomous dynamics,” Phys. Rep. 542, 297–368 (2014).
  * Kantz and Schreiber (2003) H. Kantz and T. Schreiber,  _Nonlinear Time Series Analysis_, 2nd ed. (Cambridge University Press, Cambridge, UK, 2003).
  * Boccaletti  _et al._ (2006) S. Boccaletti, V. Latora, Y. Moreno, M. Chavez, and D.-U. Hwang, “Complex networks: Structure and dynamics,” Phys. Rep. 424, 175–308 (2006).
  * Zou  _et al._ (2019) Y. Zou, R. V. Donner, N. Marwan, J. F. Donges, and J. Kurths, “Complex network approaches to nonlinear time series analysis,” Phys. Rep. 787, 1–97 (2019).
  * Amigó (2010) J. Amigó,  _Permutation complexity in dynamical systems: ordinal patterns, permutation entropy and all that_ (Springer Science & Business Media, Berlin Heidelberg, 2010).
  * Kwasniok and Smith (2004) F. Kwasniok and L. A. Smith, “Real-time construction of optimized predictors from data streams,” Phys. Rev. Lett. 92, 164101 (2004).
  * Mignan (2014) A. Mignan, “The debate on the prognostic value of earthquake foreshocks: A meta-analysis,” Sci. Rep. 4, 4099 (2014).
  * Mignan  _et al._ (2021) A. Mignan, G. Ouillon, D. Sornette, and F. Freund, “Global earthquake forecasting system (GEFS): The challenges ahead,” Eur. Phys. J. ST 230, 473–490 (2021).
  * Nyquist (1928) H. Nyquist, “Certain topics in telegraph transmission theory,” Trans. AIEE 47, 617–644 (1928).
  * Shannon (1949) C. E. Shannon, “Communication in the presence of noise,” Proc. IRE 37, 10–21 (1949).
  * Jerri (1977) A. J. Jerri, “The Shannon sampling theorem—its various extensions and applications: A tutorial review,” Proc. IEEE 65, 1565–1596 (1977).
  * Bialonski, Horstmann, and Lehnertz (2010) S. Bialonski, M. Horstmann, and K. Lehnertz, “From brain to earth and climate systems: Small-world interaction networks or not?” Chaos 20, 013134 (2010).
  * Porz, Kiel, and Lehnertz (2014) S. Porz, M. Kiel, and K. Lehnertz, “Can spurious indications for phase synchronization due to superimposed signals be avoided?” Chaos 24, 033112 (2014).
  * Effern  _et al._ (2000) A. Effern, K. Lehnertz, T. Schreiber, T. Grunwald, P. David, and C. E. Elger, “Nonlinear denoising of transient signals with application to event-related potentials,” Physica D 140, 257–266 (2000).
  * Jánosi and Tél (1994) I. M. Jánosi and T. Tél, “Time-series analysis of transient chaos,” Phys. Rev. E 49, 2756–2763 (1994).
  * Dhamala, Lai, and Kostelich (2001) M. Dhamala, Y.-C. Lai, and E. J. Kostelich, “Analyses of transient chaotic time series,” Phys. Rev. E 64, 056207 (2001).
  * Andrzejak, Ledberg, and Deco (2006) R. G. Andrzejak, A. Ledberg, and G. Deco, “Detecting event-related time-dependent directional couplings,” New J. Phys. 8, 6 (2006).
  * Łeski and Wójcik (2008) S. Łeski and D. K. Wójcik, “Inferring coupling strength from event-related dynamics,” Phys. Rev. E 78, 41918–41927 (2008).
  * Komalapriya  _et al._ (2008) C. Komalapriya, M. Thiel, M. C. Romano, N. Marwan, U. Schwarz, and J. Kurths, “Reconstruction of a system’s dynamics from short trajectories,” Phys. Rev. E 78, 066217 (2008).
  * Wagner, Fell, and Lehnertz (2010) T. Wagner, J. Fell, and K. Lehnertz, “The detection of transient directional couplings based on phase synchronization,” New J. Phys. 12, 053031 (2010).
  * Martini  _et al._ (2011) M. Martini, T. A. Kranz, T. Wagner, and K. Lehnertz, “Inferring directional interactions from transient signals with symbolic transfer entropy,” Phys. Rev. E 83, 011919 (2011).
  * Ma  _et al._ (2014) H. Ma, T. Zhou, K. Aihara, and L. Chen, “Predicting time series from short-term high-dimensional data,” Int.. J. Bifurcation Chaos 24, 1430033 (2014).
  * Wang, Lai, and Grebogi (2016) W.-X. Wang, Y.-C. Lai, and C. Grebogi, “Data based identification and prediction of nonlinear and complex dynamical systems,” Phys. Rep. 644, 1–76 (2016).
  * Kaneko (1990) K. Kaneko, “Supertransients, spatiotemporal intermittency and stability of fully developed spatiotemporal chaos,” Phys. Lett. A 149, 105–112 (1990).
  * Tél and Lai (2008) T. Tél and Y.-C. Lai, “Chaotic transients in spatially extended systems,” Phys. Rep. 460, 245–275 (2008).
  * Meyer-Ortmanns (2023) H. Meyer-Ortmanns, “Heteroclinic networks for brain dynamics,” Front. Netw. Physiol. 3, 1276401 (2023).
  * Ansmann, Lehnertz, and Feudel (2016) G. Ansmann, K. Lehnertz, and U. Feudel, “Self-induced switchings between multiple space-time patterns on complex networks of excitable units,” Phys. Rev. X 6, 011030 (2016).
  * Nandan and Koseska (2023) A. Nandan and A. Koseska, “Non-asymptotic transients away from steady states determine cellular responsiveness to dynamic spatial-temporal signals,” PLoS Comput. Biol. 19, e1011388 (2023).
  * Boltzmann (1877) L. Boltzmann,  _Über die Beziehung zwischen dem zweiten Hauptsatze des mechanischen Wärmetheorie und der Wahrscheinlichkeitsrechnung, respective den Sätzen über das Wärmegleichgewicht_ (K.k. Hof-und Staatsdruckerei, Vienna, Austria, 1877).
  * Birkhoff (1931) G. D. Birkhoff, “Proof of the ergodic theorem,” Proc. Natl. Acad. Sci. 17, 656–660 (1931).
  * Priestley (1988) M. B. Priestley,  _Nonlinear and Non-Stationary Time Series Analysis_ (Academic Press, London, 1988).
  * Prabhu (2014) K. M. M. Prabhu,  _Window functions and their applications in signal processing_ (Taylor & Francis, Boca Raton, USA, 2014).
  * Kitagawa (1987) G. Kitagawa, “Non-Gaussian state—space modeling of nonstationary time series,” J. Am. Stat. Assoc. 82, 1032–1041 (1987).
  * Dahlhaus (1997) R. Dahlhaus, “Fitting time series models to nonstationary processes,” Ann. Stat. 25, 1–37 (1997).
  * Huang  _et al._ (1998) N. E. Huang, Z. Shen, S. R. Long, M. C. Wu, H. H. Shih, Q. Zheng, N.-C. Yen, C. C. Tung, and H. H. Liu, “The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis,” Proc. Roy. Soc. London. Series A: Mathematical, Physical and Engineering Sciences 454, 903–995 (1998).
  * Verdes  _et al._ (2001) P. Verdes, P. Granitto, H. Navone, and H. Ceccatto, “Nonstationary time-series analysis: Accurate reconstruction of driving forces,” Phys. Rev. Lett. 87, 124101 (2001).
  * Kantelhardt  _et al._ (2002) J. W. Kantelhardt, S. A. Zschiegner, E. Koscielny-Bunde, S. Havlin, A. Bunde, and H. E. Stanley, “Multifractal detrended fluctuation analysis of nonstationary time series,” Physica A: Statistical Mechanics and its Applications 316, 87–114 (2002).
  * Fukuda, Stanley, and Amaral (2004) K. Fukuda, H. E. Stanley, and L. A. N. Amaral, “Heuristic segmentation of a nonstationary time series,” Phys. Rev. E 69, 021108 (2004).
  * Wu  _et al._ (2007) Z. Wu, N. E. Huang, S. R. Long, and C.-K. Peng, “On the trend, detrending, and variability of nonlinear and nonstationary time series,” Proc. Natl. Acad. Sci. 104, 14889–14894 (2007).
  * Lehnertz  _et al._ (2017) K. Lehnertz, C. Geier, T. Rings, and K. Stahn, “Capturing time-varying brain dynamics,” EPJ Nonlin. Biomed. Phys. 5, 2 (2017).
  * Podobnik and Stanley (2008) B. Podobnik and H. E. Stanley, “Detrended cross-correlation analysis: a new method for analyzing two nonstationary time series,” Phys. Eev. Lett. 100, 084102 (2008).
  * Rhif  _et al._ (2019) M. Rhif, A. Ben Abbes, I. R. Farah, B. Martínez, and Y. Sang, “Wavelet transform application for/in non-stationary time-series analysis: A review,” Applied Sciences 9, 1345 (2019).
  * Hegger  _et al._ (2000) R. Hegger, H. Kantz, L. Matassini, and T. Schreiber, “Coping with non-stationarity by overembedding,” Phys. Rev. Lett. 84, 4092–4095 (2000).
  * Verdes, Granitto, and Ceccatto (2006) P. Verdes, P. Granitto, and H. Ceccatto, “Overembedding method for modeling nonstationary systems,” Phys. Rev. Lett. 96, 118701 (2006).
  * De Domenico and Latora (2010) M. De Domenico and V. Latora, “Fast detection of nonlinearity and nonstationarity in short and noisy time series,” Europhys. Lett. 91, 30005 (2010).
  * Manuca and Savit (1996) R. Manuca and R. Savit, “Stationarity and nonstationarity in time series analysis,” Physica D: Nonlinear Phenomena 99, 134–161 (1996).
  * Kennel (1997) M. B. Kennel, “Statistical test for dynamical nonstationarity in observed time-series data,” Phys. Rev. E 56, 316–321 (1997).
  * Gao (2001) J. Gao, “Detecting nonstationarity and state transitions in a time series,” Phys. Rev. E 63, 066202 (2001).
  * Rieke  _et al._ (2002) C. Rieke, K. Sternickel, R. G. Andrzejak, C. E. Elger, P. David, and K. Lehnertz, “Measuring nonstationarity by analyzing the loss of recurrence in dynamical systems,” Phys. Rev. Lett. 88, 244102 (2002).
  * Rieke  _et al._ (2004) C. Rieke, R. G. Andrzejak, F. Mormann, and K. Lehnertz, “Improved statistical test for nonstationarity using recurrence time statistics,” Phys. Rev. E 69, 046111 (2004).
  * Facchini, Kantz, and Tiezzi (2005) A. Facchini, H. Kantz, and E. Tiezzi, “Recurrence plot analysis of nonstationary data: the understanding of curved patterns,” Phys. Rev. E 72, 021915 (2005).
  * Chen and Yang (2012) Y. Chen and H. Yang, “Multiscale recurrence analysis of long-term nonlinear and nonstationary time series,” Chaos, Solitons & Fractals 45, 978–987 (2012).
  * Schreiber (1997) T. Schreiber, “Detecting and analysing nonstationarity in a time series using nonlinear cross predictions,” Phys. Rev. Lett. 78, 843 (1997).
  * Kennel and Mees (2000) M. B. Kennel and A. I. Mees, “Testing for general dynamical stationarity with a symbolic data compression technique,” Phys. Rev. E 61, 2563 (2000).
  * Robinson (1994) P. M. Robinson, “Efficient tests of nonstationary hypotheses,” J. Am. Stat. Assoc. 89, 1420–1437 (1994).
  * Witt, Kurths, and Pikovsky (1998) A. Witt, J. Kurths, and A. Pikovsky, “Testing stationarity in time series,” Phys. Rev. E 58, 1800–1810 (1998).
  * Champeney (1973) D. C. Champeney,  _Fourier Transforms and their physical applications_ (Academic Press, Cambridge, MA, USA, 1973).
  * Brody  _et al._ (1981) T. A. Brody, J. Flores, J. B. French, P. A. Mello, A. Pandey, and S. S. M. Wong, “Random-matrix physics: Spectrum and strength fluctuations,” Rev. Mod. Phys. 53, 385–479 (1981).
  * Haykin (1983) S. Haykin, ed.,  _Nonlinear methods of spectral analysis_ (Springer, Berlin, Heidelberg, 1983).
  * Grassberger, Schreiber, and Schaffrath (1991) P. Grassberger, T. Schreiber, and C. Schaffrath, “Nonlinear time sequence analysis,” Int. J. Bifurcation Chaos Appl. Sci. Eng. 1, 521 (1991).
  * Abarbanel  _et al._ (1993) H. D. I. Abarbanel, R. Brown, J. J. Sidorowich, and L. S. Tsimring, “The analysis of observed chaotic data in physical systems,” Rev. Mod. Phys. 65, 1331 (1993).
  * Honerkamp (1993) J. Honerkamp,  _Stochastic Dynamical Systems: Concepts, Numerical Methods, Data Analysis_ (Wiley-VCH, New York, 1993).
  * Kaplan and Glass (1995) D. Kaplan and L. Glass,  _Understanding Nonlinear Dynamics_ (Springer, New York, 1995).
  * Percival and Walden (2000) D. B. Percival and A. T. Walden,  _Wavelet methods for time series analysis_ , Vol. 4 (Cambridge University Press, Cambridge, UK, 2000).
  * Pikovsky, Rosenblum, and Kurths (2001) A. S. Pikovsky, M. G. Rosenblum, and J. Kurths,  _Synchronization: A universal concept in nonlinear sciences_ (Cambridge University Press, Cambridge, UK, 2001).
  * Daw, Finney, and Tracy (2003) C. Daw, C. Finney, and E. Tracy, “A review of symbolic analysis of experimental data.” Rev. Sci. Instrum. 74, 915–930 (2003).
  * Reinsel (2003) G. C. Reinsel,  _Elements of multivariate time series analysis_ , 2nd ed. (Springer, New York, 2003).
  * Keller and Sinn (2005) K. Keller and M. Sinn, “Ordinal analysis of time series,” Physica A: Statistical Mechanics and its Applications 356, 114–120 (2005).
  * Lütkepohl (2005) H. Lütkepohl,  _New introduction to multiple time series analysis_ (Springer Science & Business Media, Berlin, Heidelberg, Germany, 2005).
  * Hlaváčková-Schindler  _et al._ (2007) K. Hlaváčková-Schindler, M. Paluš, M. Vejmelka, and J. Bhattacharya, “Causality detection based on information-theoretic approaches in time series analysis,” Phys. Rep. 441, 1–46 (2007).
  * Marwan  _et al._ (2007) N. Marwan, M. C. Romano, M. Thiel, and J. Kurths, “Recurrence plots for the analysis of complex systems,” Phys. Rep. 438, 237–329 (2007).
  * Lacasa  _et al._ (2008) L. Lacasa, B. Luque, F. Ballesteros, J. Luque, and J. C. Nuno, “From time series to complex networks: The visibility graph,” Proc. Natl. Acad. Sci. U.S.A. 105, 4972–4975 (2008).
  * Anderson (2011) T. W. Anderson,  _The statistical analysis of time series_ (John Wiley & Sons, Hoboken, New Jersey, 2011).
  * Friedrich  _et al._ (2011) R. Friedrich, J. Peinke, M. Sahimi, and M. R. R. Tabar, “Approaching complexity by stochastic methods: From biological systems to turbulence,” Phys. Rep. 506, 87–162 (2011).
  * Von Toussaint (2011) U. Von Toussaint, “Bayesian inference in physics,” Rev. Mod. Phys. 83, 943 (2011).
  * Bradley and Kantz (2015) E. Bradley and H. Kantz, “Nonlinear time-series analysis revisited,” Chaos: An Interdisciplinary Journal of Nonlinear Science 25, 097610 (2015).
  * Webber, Jr. and Marwan (2015) C. L. Webber, Jr. and N. Marwan,  _Recurrence Quantification Analysis – Theory and Best Practices_ (Springer, Cham, Switzerland, 2015).
  * Stankovski  _et al._ (2017) T. Stankovski, T. Pereira, P. V. E. McClintock, and A. Stefanovska, “Coupling functions: Universal insights into dynamical interaction mechanisms,” Rev. Mod. Phys. 89, 045001 (2017).
  * Tabar (2019) M. R. R. Tabar,  _Analysis and Data-Based Reconstruction of Complex Nonlinear Dynamical Systems: Using the Methods of Stochastic Processes_ (Springer International Publishin, Cham, Switzerland, 2019).
  * Hamilton (2020) J. D. Hamilton,  _Time series analysis_ (Princeton University Press, 2020).
  * Edinburgh, Eglen, and Ercole (2021) T. Edinburgh, S. J. Eglen, and A. Ercole, “Causality indices for bivariate time series data: A comparative review of performance,” Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 083111 (2021).
  * Datseris and Parlitz (2022) G. Datseris and U. Parlitz,  _Nonlinear dynamics: a concise introduction interlaced with code_ (Springer International Publishing, Cham, Switzerland, 2022).
  * Fokianos  _et al._ (2022) K. Fokianos, R. Fried, Y. Kharin, and V. Voloshko, “Statistical analysis of multivariate discrete-valued time series,” J. Multivar. Anal. 188, 104805 (2022).
  * Nikakhtar  _et al._ (2023) F. Nikakhtar, L. Parkavousi, M. Sahimi, M. R. R. Tabar, U. Feudel, and K. Lehnertz, “Data-driven reconstruction of stochastic dynamical equations based on statistical moments,” New J. Phys. 25, 083025 (2023).
  * Tabar  _et al._ (2024) M. R. R. Tabar, F. Nikakhtar, L. Parkavousi, A. Akhshi, U. Feudel, and K. Lehnertz, “Revealing higher-order interactions in high-dimensional complex systems: A data-driven approach,” Phys. Rev. X 14, 011050 (2024).
  * Packard  _et al._ (1980) N. H. Packard, J. P. Crutchfield, J. D. Farmer, and R. S. Shaw, “Geometry from a time series,” Phys. Rev. Lett. 45, 712–716 (1980).
  * Takens (1981) F. Takens, “Detecting strange attractors in turbulence,” in  _Dynamical Systems and Turbulence (Warwick 1980)_, Lecture Notes in Mathematics, Vol. 898, edited by D. A. Rand and L.-S. Young (Springer, Berlin, 1981) pp. 366–381.
  * Sauer, Yorke, and Casdagli (1991) T. Sauer, J. Yorke, and M. Casdagli, “Embedology,” J. Stat. Phys. 65, 579–616 (1991).
  * Casdagli  _et al._ (1991) M. Casdagli, S. Eubank, J. D. Farmer, and J. Gibson, “State space reconstruction in the presence of noise,” Physica D: Nonlinear Phenomena 51, 52–98 (1991).
  * Kennel, Brown, and Abarbanel (1992) M. B. Kennel, R. Brown, and H. D. I. Abarbanel, “Determining embedding dimension for phase-space reconstruction using a geometrical construction,” Phys. Rev. A 45, 3403–3411 (1992).
  * Kugiumtzis (1996) D. Kugiumtzis, “State space reconstruction parameters in the analysis of chaotic time series–the role of the time window length,” Physica D: Nonlinear Phenomena 95, 13–28 (1996).
  * Cao (1997) L. Cao, “Practical method for determining the minimum embedding dimension of a scalar time series,” Physica D 110, 43–50 (1997).
  * Cellucci, Albano, and Rapp (2003) C. J. Cellucci, A. M. Albano, and P. E. Rapp, “Comparative study of embedding methods,” Phys. Rev. E 67, 066210 (2003).
  * Vlachos and Kugiumtzis (2010) I. Vlachos and D. Kugiumtzis, “Nonuniform state-space reconstruction and coupling detection,” Phys. Rev. E 82, 016207 (2010).
  * Krämer  _et al._ (2021) K.-H. Krämer, G. Datseris, J. Kurths, I. Z. Kiss, J. L. Ocampo-Espindola, and N. Marwan, “A unified and automated approach to attractor reconstruction,” New J. Physics 23, 033017 (2021).
  * Dakos  _et al._ (2008) V. Dakos, M. Scheffer, E. H. van Nes, V. Brovkin, V. Petoukhov, and H. Held, “Slowing down as an early warning signal for abrupt climate change,” Proc. Natl. Acad. Sci. U.S.A. 105, 14308–14312 (2008).
  * Scheffer  _et al._ (2009) M. Scheffer, J. Bascompte, W. A. Brock, V. Brovkin, S. R. Carpenter, V. Dakos, H. Held, E. H. van Nes, M. Rietkerk, and G. Sugihara, “Early-warning signals for critical transitions,” Nature 461, 53–59 (2009).
  * Scheffer  _et al._ (2012) M. Scheffer, S. R. Carpenter, T. M. Lenton, J. Bascompte, W. Brock, V. Dakos, J. van de Koppel, I. A. van de Leemput, S. A. Levin, E. H. van Nes, M. Pascual, and J. Vandermeer, “Anticipating critical transitions,” Science 338, 344–348 (2012).
  * Dai  _et al._ (2012) L. Dai, D. Vorselen, K. S. Korolev, and J. Gore, “Generic indicators for loss of resilience before a tipping point leading to population collapse,” Science 336, 1175–1177 (2012).
  * Lenton  _et al._ (2012) T. Lenton, V. Livina, V. Dakos, E. Van Nes, and M. Scheffer, “Early warning of climate tipping points from critical slowing down: comparing methods to improve robustness,” Phil. Trans. Roy. Soc. A: Mathematical, Physical and Engineering Sciences 370, 1185–1204 (2012).
  * Bury, Bauch, and Anand (2020) T. M. Bury, C. T. Bauch, and M. Anand, “Detecting and distinguishing tipping points using spectral early warning signals,” J. Roy. Soc. Interface 17, 20200482 (2020).
  * Kubo (1966) R. Kubo, “The fluctuation-dissipation theorem,” Rep. Prog. Phys. 29, 255 (1966).
  * Cotilla-Sanchez, Hines, and Danforth (2012) E. Cotilla-Sanchez, P. D. Hines, and C. M. Danforth, “Predicting critical transitions from time series synchrophasor data,” IEEE Trans. Smart Grid 3, 1832–1840 (2012).
  * Dakos  _et al._ (2012) V. Dakos, S. R. Carpenter, W. A. Brock, A. M. Ellison, V. Guttal, A. R. Ives, S. Kéfi, V. Livina, D. A. Seekell, E. H. van Nes, and M. Scheffer, “Methods for detecting early warnings of critical transitions in time series illustrated using simulated ecological data,” PLOS ONE 7, 1–20 (2012).
  * van de Leemput  _et al._ (2014) I. A. van de Leemput, M. Wichers, A. O. Cramer, D. Borsboom, F. Tuerlinckx, P. Kuppens, E. H. van Nes, W. Viechtbauer, E. J. Giltay, S. H. Aggen, _et al._ , “Critical slowing down as early warning for the onset and termination of depression,” Proc. Natl. Acad. Sci. U.S.A. 111, 87–92 (2014).
  * Meisel  _et al._ (2015) C. Meisel, A. Klaus, C. Kuehn, and D. Plenz, “Critical slowing down governs the transition to neuron spiking,” PLoS Comput. Biol. 11, e1004097 (2015).
  * Perry, Livina, and Niewczas (2015) M. Perry, V. Livina, and P. Niewczas, “Tipping point analysis of cracking in reinforced concrete,” Smart Mater. Struct. 25, 015027 (2015).
  * Nazarimehr  _et al._ (2020) F. Nazarimehr, S. Jafari, M. Perc, and J. C. Sprott, “Critical slowing down indicators,” Europhys. Lett. 132, 18001 (2020).
  * Brookes  _et al._ (2021) P. Brookes, G. Tancredi, A. D. Patterson, J. Rahamim, M. Esposito, T. K. Mavrogordatos, P. J. Leek, E. Ginossar, and M. H. Szymanska, “Critical slowing down in circuit quantum electrodynamics,” Sci. Adv. 7, eabe9492 (2021).
  * George, Kachhara, and Ambika (2021) S. V. George, S. Kachhara, and G. Ambika, “Early warning signals for critical transitions in complex systems,” Phys. Scr. 98, 072002 (2021).
  * Southall  _et al._ (2021) E. Southall, T. S. Brett, M. J. Tildesley, and L. Dyson, “Early warning signals of infectious disease transitions: a review,” J. Roy. Soc. Interface 18, 20210555 (2021).
  * Dablander  _et al._ (2022) F. Dablander, A. Pichler, A. Cika, and A. Bacilieri, “Anticipating critical transitions in psychological systems using early warning signals: Theoretical and practical considerations.” Psychol. Methods 28, 765–790 (2022).
  * Deb  _et al._ (2022) S. Deb, S. Bhandary, S. K. Sinha, M. K. Jolly, and P. S. Dutta, “Identifying critical transitions in complex diseases,” J. Biosci. 47, 25 (2022).
  * Ditlevsen and Ditlevsen (2023) P. Ditlevsen and S. Ditlevsen, “Warning of a forthcoming collapse of the Atlantic meridional overturning circulation,” Nat. Commun. 14, 1–12 (2023).
  * Mathevet  _et al._ (2024) R. Mathevet, P. Marchou, C. Fabre, N. Lamrani, and N. Combe, “Coriolis acceleration and critical slowing-down: A quantitative laboratory experiment,” Am. J. Physics 92, 100–107 (2024).
  * Ditlevsen and Johnsen (2010) P. D. Ditlevsen and S. J. Johnsen, “Tipping points: early warning and wishful thinking,” Geophys. Res. Lett. 37, L19703 (2010).
  * Boettiger and Hastings (2012a) C. Boettiger and A. Hastings, “Early warning signals and the prosecutor’s fallacy,” Proc Roy Soc. B: Biol. Sci. 279, 4734–4739 (2012a).
  * Boettiger and Hastings (2013) C. Boettiger and A. Hastings, “No early warning signals for stochastic transitions: insights from large deviation theory,” Proc. Roy. Soc. B: Biol. Sci. 280, 20131372 (2013).
  * Boettiger, Ross, and Hastings (2013) C. Boettiger, N. Ross, and A. Hastings, “Early warning signals: the charted and uncharted territories,” Theor. Ecol. 6, 255–264 (2013).
  * Kéfi  _et al._ (2013) S. Kéfi, V. Dakos, M. Scheffer, E. H. Van Nes, and M. Rietkerk, “Early warning signals also precede non-catastrophic transitions,” Oikos 122, 641–648 (2013).
  * Guttal, Jayaprakash, and Tabbaa (2013) V. Guttal, C. Jayaprakash, and O. P. Tabbaa, “Robustness of early warning signals of regime shifts in time-delayed ecological models,” Theor. Ecol. 6, 271–283 (2013).
  * Dakos  _et al._ (2015) V. Dakos, S. R. Carpenter, E. H. van Nes, and M. Scheffer, “Resilience indicators: prospects and limitations for early warnings of regime shifts,” Phil. Trans. R. Soc. B: Biol. Sci. 370, 20130263 (2015).
  * Dai, Korolev, and Gore (2015) L. Dai, K. S. Korolev, and J. Gore, “Relation between stability and resilience determines the performance of early warning signals under different environmental drivers,” Proc. Natl. Acad. Sci. U.S.A. 112, 10056–10061 (2015).
  * Diks, Hommes, and Wang (2015) C. Diks, C. Hommes, and J. Wang, “Critical slowing down as an early warning signal for financial crises?” Empirical Economics , 1–28 (2015).
  * Wagner and Eisenman (2015) T. J. Wagner and I. Eisenman, “False alarms: How early warning signals falsely predict abrupt sea ice loss,” Geophys. Res. Lett. 42, 10–333 (2015).
  * Zhang, Kuehn, and Hallerberg (2015) X. Zhang, C. Kuehn, and S. Hallerberg, “Predictability of critical transitions,” Phys. Rev. E 92, 052905 (2015).
  * Guttal  _et al._ (2016) V. Guttal, S. Raghavendra, N. Goel, and Q. Hoarau, “Lack of critical slowing down suggests that financial meltdowns are not critical transitions, yet rising variability could signal systemic risk,” PloS one 11, e0144198 (2016).
  * Gsell  _et al._ (2016) A. S. Gsell, U. Scharfenberger, D. Özkundakci, A. Walters, L.-A. Hansson, A. B. Janssen, P. Nõges, P. C. Reid, D. E. Schindler, E. Van Donk, _et al._ , “Evaluating early-warning indicators of critical transitions in natural aquatic ecosystems,” Proc. Natl. Acad. Sci.. 113, E8089–E8095 (2016).
  * Milanowski and Suffczynski (2016) P. Milanowski and P. Suffczynski, “Seizures start without common signatures of critical transition,” Int. J. Neural Syst. 26, 1650053 (2016).
  * Dutta, Sharma, and Abbott (2018) P. S. Dutta, Y. Sharma, and K. C. Abbott, “Robustness of early warning signals for catastrophic and non-catastrophic transitions,” Oikos 127, 1251–1263 (2018).
  * Wen, Ciamarra, and Cheong (2018) H. Wen, M. P. Ciamarra, and S. A. Cheong, “How one might miss early warning signals of critical transitions in time series data: A systematic study of two major currency pairs,” PloS one 13, e0191439 (2018).
  * Romano and Kuehn (2018) F. Romano and C. Kuehn, “Analysis and predictability of tipping points with leading-order nonlinear term,” Int. J. Bifurcation Chaos 28, 1850103 (2018).
  * Arumugam  _et al._ (2019) R. Arumugam, S. Sarkar, T. Banerjee, S. Sinha, and P. S. Dutta, “Dynamic environment-induced multistability and critical transition in a metacommunity ecosystem,” Phys. Rev. E 99, 032216 (2019).
  * Clements, McCarthy, and Blanchard (2019) C. F. Clements, M. A. McCarthy, and J. L. Blanchard, “Early warning signals of recovery in complex systems,” Nat. Commun. 10, 1681 (2019).
  * Gatfaoui and De Peretti (2019) H. Gatfaoui and P. De Peretti, “Flickering in information spreading precedes critical transitions in financial markets,” Sci. Rep. 9, 5671 (2019).
  * Jäger and Füllsack (2019) G. Jäger and M. Füllsack, “Systematically false positives in early warning signal analysis,” PloS one 14, e0211072 (2019).
  * Wilkat, Rings, and Lehnertz (2019) T. Wilkat, T. Rings, and K. Lehnertz, “No evidence for critical slowing down prior to human epileptic seizures,” Chaos 29, 091104 (2019).
  * Marconi  _et al._ (2020) M. Marconi, C. Métayer, A. Acquaviva, J. Boyer, A. Gomel, T. Quiniou, C. Masoller, M. Giudici, and J. Tredicce, “Testing critical slowing down as a bifurcation indicator in a low-dissipation dynamical system,” Phys. Rev. Lett. 125, 134102 (2020).
  * van der Bolt, van Nes, and Scheffer (2021) B. van der Bolt, E. H. van Nes, and M. Scheffer, “No warning for slow transitions,” J. Roy. Soc. Interface 18, 20200935 (2021).
  * Lapeyrolerie and Boettiger (2023) M. Lapeyrolerie and C. Boettiger, “Limits to ecological forecasting: Estimating uncertainty for critical transitions with deep learning,” Methods Ecol. Evol 14, 785–798 (2023).
  * O’Brien  _et al._ (2023) D. A. O’Brien, S. Deb, G. Gal, S. J. Thackeray, P. S. Dutta, S.-i. S. Matsuzaki, L. May, and C. F. Clements, “Early warning signals have limited applicability to empirical lake data,” Nat. Commun. 14, 7942 (2023).
  * Proverbio, Skupin, and Gonçalves (2023) D. Proverbio, A. Skupin, and J. Gonçalves, “Systematic analysis and optimization of early warning signals for critical transitions using distribution data,” iScience 26, 107156 (2023).
  * Grebogi, Ott, and Yorke (1983) C. Grebogi, E. Ott, and J. A. Yorke, “Crises, sudden changes in chaotic attractors, and transient chaos,” Physica D 7, 181–200 (1983).
  * Osinga and Feudel (2000) H. M. Osinga and U. Feudel, “Boundary crisis in quasiperiodically forced systems,” Physica D: Nonlinear Phenomena 141, 54–64 (2000).
  * Ott and Summerer (1994) E. Ott and J. C. Summerer, “Blowout bifurcations: the occurence of riddled basins and on-off intermittency,” Phys. Lett. A 188, 39–47 (1994).
  * Zhang  _et al._ (2020) Y. Zhang, Z. G. Nicolaou, J. D. Hart, R. Roy, and A. E. Motter, “Critical switching in globally attractive chimeras,” Phys. Rev. X 10, 011044 (2020).
  * Kuehn, Zschaler, and Gross (2015) C. Kuehn, G. Zschaler, and T. Gross, “Early warning signs for saddle-escape transitions in complex networks,” Sci. Rep. 5, 13190 (2015).
  * Koseska, Volkov, and Kurths (2013) A. Koseska, E. Volkov, and J. Kurths, “Oscillation quenching mechanisms: Amplitude vs. oscillation death,” Phys. Rep. 531, 173 – 199 (2013).
  * Zou  _et al._ (2021) W. Zou, D. Senthilkumar, M. Zhan, and J. Kurths, “Quenching, aging, and reviving in coupled dynamical networks,” Phys. Rep. 931, 1–72 (2021).
  * Boccaletti  _et al._ (2016) S. Boccaletti, J. Almendral, S. Guan, I. Leyva, Z. Liu, I. Sendiña-Nadal, Z. Wang, and Y. Zou, “Explosive transitions in complex networks’ structure and dynamics: Percolation and synchronization,” Phys. Rep. 660, 1–94 (2016).
  * Kuehn and Bick (2021) C. Kuehn and C. Bick, “A universal route to explosive phenomena,” Sci. Adv. 7, eabe3824 (2021).
  * Mormann  _et al._ (2003) F. Mormann, R. Andrzejak, T. Kreuz, C. Rieke, P. David, C. E. Elger, and K. Lehnertz, “Automated detection of a preseizure state based on a decrease in synchronization in intracranial electroencephalogram recordings from epilepsy patients,” Phys. Rev. E 67, 021912 (2003).
  * Mormann  _et al._ (2005) F. Mormann, T. Kreuz, C. Rieke, R. G. Andrzejak, A. Kraskov, P. David, C. E. Elger, and K. Lehnertz, “On the predictability of epileptic seizures,” Clin. Neurophysiol. 116, 569–587 (2005).
  * Winterhalder  _et al._ (2006) M. Winterhalder, B. Schelter, T. Maiwald, A. Brandt, A. Schad, A. Schulze-Bonhage, and J. Timmer, “Spatio-temporal patient-individual assessment of synchronization changes for epileptic seizure prediction,” Clin. Neurophysiol. 117, 2399–2413 (2006).
  * Kuhlmann  _et al._ (2010) L. Kuhlmann, D. Freestone, A. L. Lai, A. N. Burkitt, K. Fuller, D. Grayden, L. Seiderer, S. Vogrin, I. M. Y. Mareels, and M. J. Cook, “Patient-specific bivariate-synchrony-based seizure prediction for short prediction horizons,” Epilepsy Res. 91, 214–231 (2010).
  * Ray (2004) A. Ray, “Symbolic dynamic analysis of complex systems for anomaly detection,” Signal Process. 84, 1115–1130 (2004).
  * Chin, Ray, and Rajagopalan (2005) S. C. Chin, A. Ray, and V. Rajagopalan, “Symbolic time series analysis for anomaly detection: A comparative evaluation,” Signal Process. 85, 1859–1868 (2005).
  * Lehnertz and Dickten (2015) K. Lehnertz and H. Dickten, “Assessing directionality and strength of coupling through symbolic analysis: an application to epilepsy patients,” Phil. Trans. R. Soc. A 373, 20140094 (2015).
  * Liu  _et al._ (2015) R. Liu, P. Chen, K. Aihara, and L. Chen, “Identifying early-warning signals of critical transitions with strong noise by dynamical network markers,” Sci. Rep. 5, 17501 (2015).
  * Peng  _et al._ (2019) X. Peng, M. Small, Y. Zhao, and J. M. Moore, “Detecting and predicting tipping points,” Int. J. Bifurcation Chaos 29, 1930022 (2019).
  * Rings  _et al._ (2019) T. Rings, M. Mazarei, A. Akhshi, C. Geier, M. R. R. Tabar, and K. Lehnertz, “Traceability and dynamical resistance of precursor of extreme events,” Sci. Rep. 9, 1744 (2019).
  * Rings, von Wrede, and Lehnertz (2019) T. Rings, R. von Wrede, and K. Lehnertz, “Precursors of seizures due to specific spatial-temporal modifications of evolving large-scale epileptic brain networks,” Sci. Rep. 9, 10623 (2019).
  * Ludescher  _et al._ (2021) J. Ludescher, M. Martin, N. Boers, A. Bunde, C. Ciemer, J. Fan, S. Havlin, M. Kretschmer, J. Kurths, J. Runge, _et al._ , “Network-based forecasting of climate phenomena,” Proc. Natl. Acad. Sci. 118, e1922872118 (2021).
  * Mittal and Gupta (2017) K. Mittal and S. Gupta, “Topological characterization and early detection of bifurcations and chaos in complex systems using persistent homology,” Chaos: An Interdisciplinary Journal of Nonlinear Science 27, 051102 (2017).
  * Syed Musa  _et al._ (2021) S. M. S. Syed Musa, M. S. Md Noorani, F. Abdul Razak, M. Ismail, M. A. Alias, and S. I. Hussain, “Using persistent homology as preprocessing of early warning signals for critical transition in flood,” Sci. Rep. 11, 7234 (2021).
  * Ghil and Sciamarella (2023) M. Ghil and D. Sciamarella, “Dynamical systems, algebraic topology and the climate sciences,” Nonlin. Proc. Geophys. 30, 399–434 (2023).
  * Giacomini and Rossi (2009) R. Giacomini and B. Rossi, “Detecting and predicting forecast breakdowns,” Rev. Econ. Stud. 76, 669–705 (2009).
  * Ghalati  _et al._ (2019) P. F. Ghalati, S. S. Samal, J. S. Bhat, R. Deisz, G. Marx, and A. Schuppert, “Critical transitions in intensive care units: a sepsis case study,” Sci. Rep. 9, 12888 (2019).
  * Liu  _et al._ (2012) R. Liu, M. Li, Z.-P. Liu, J. Wu, L. Chen, and K. Aihara, “Identifying critical transitions and their leading biomolecular networks in complex diseases,” Sci. Rep. 2, 813 (2012).
  * Meng  _et al._ (2020) J. Meng, J. Fan, J. Ludescher, A. Agarwal, X. Chen, A. Bunde, J. Kurths, and H. J. Schellnhuber, “Complexity-based approach for El Niño magnitude forecasting before the spring predictability barrier,” Proc. Nat. Acad. Sci. 117, 177–183 (2020).
  * Pavithran, Unni, and Sujith (2021) I. Pavithran, V. R. Unni, and R. Sujith, “Critical transitions and their early warning signals in thermoacoustic systems,” Eur. Phys. J. Spec. Top. 230, 3411–3432 (2021).
  * Tirabassi and Masoller (2023) G. Tirabassi and C. Masoller, “Entropy-based early detection of critical transitions in spatial vegetation fields,” Proc. Natl. Acad. Sci. 120, e2215667120 (2023).
  * Deb and Dutta (2024) S. Deb and P. S. Dutta, “Critical transitions in spatial systems induced by ornstein–uhlenbeck noise: spatial mutual information as a precursor,” Proc. R. Soc. A.: Mathematical, Physical and Engineering Sciences 480, 20230594 (2024).
  * Barter  _et al._ (2021) E. Barter, A. Brechtel, B. Drossel, and T. Gross, “A closed form for jacobian reconstruction from time series and its application as an early warning signal in network dynamics,” Proc. Roy. Soc. A 477, 20200742 (2021).
  * Franzke (2013) C. L. Franzke, “Predictions of critical transitions with non-stationary reduced order models,” Physica D: Nonlinear Phenomena 262, 35–47 (2013).
  * Kwasniok (2013) F. Kwasniok, “Predicting critical transitions in dynamical systems from time series using nonstationary probability density modeling,” Phys. Rev. E 88, 052917 (2013).
  * Kwasniok (2015) F. Kwasniok, “Forecasting critical transitions using data-driven nonstationary dynamical modeling,” Phys. Rev. E 92, 062928 (2015).
  * Kwasniok (2018) F. Kwasniok, “Detecting, anticipating, and predicting critical transitions in spatially extended systems,” Chaos: An Interdisciplinary Journal of Nonlinear Science 28, 033614 (2018).
  * Din, Liang, and Zhou (2018) A. Din, J. Liang, and T. Zhou, “Detecting critical transitions in the case of moderate or strong noise by binomial moments,” Phys. Rev. E 98, 012114 (2018).
  * Lehnertz, Zabawa, and Tabar (2018) K. Lehnertz, L. Zabawa, and M. R. R. Tabar, “Characterizing abrupt transitions in stochastic dynamics,” New J. Physics 20, 113043 (2018).
  * Arani  _et al._ (2021) B. M. Arani, S. R. Carpenter, L. Lahti, E. H. Van Nes, and M. Scheffer, “Exit time as a measure of ecological resilience,” Science 372, eaay4895 (2021).
  * Heßler and Kamps (2022) M. Heßler and O. Kamps, “Bayesian on-line anticipation of critical transitions,” New J. Phys. 24, 063021 (2022).
  * Heßler and Kamps (2023) M. Heßler and O. Kamps, “Quantifying resilience and the risk of regime shifts under strong correlated noise,” PNAS nexus 2, pgac296 (2023).
  * Lim  _et al._ (2020) S. H. Lim, L. Theo Giorgini, W. Moon, and J. S. Wettlaufer, “Predicting critical transitions in multiscale dynamical systems using reservoir computing,” Chaos: An Interdisciplinary Journal of Nonlinear Science 30, 123126 (2020).
  * Kong  _et al._ (2021) L.-W. Kong, H.-W. Fan, C. Grebogi, and Y.-C. Lai, “Machine learning prediction of critical transition and system collapse,” Phys. Rev. Res. 3, 013090 (2021).
  * Ray, Chakraborty, and Ghosh (2021) A. Ray, T. Chakraborty, and D. Ghosh, “Optimized ensemble deep learning framework for scalable forecasting of dynamics containing extreme events,” Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 111105 (2021).
  * Patel  _et al._ (2021) D. Patel, D. Canaday, M. Girvan, A. Pomerance, and E. Ott, “Using machine learning to predict statistical properties of non-stationary dynamical processes: System climate, regime transitions, and the effect of stochasticity,” Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 033149 (2021).
  * Patel and Ott (2023) D. Patel and E. Ott, “Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems,” Chaos: An Interdisciplinary Journal of Nonlinear Science 33, 023143 (2023).
  * Köglmayr and Räth (2024) D. Köglmayr and C. Räth, “Extrapolating tipping points and simulating non-stationary dynamics of complex systems using efficient machine learning,” Sci. Rep. 14, 507 (2024).
  * Shcherbakov  _et al._ (2019) R. Shcherbakov, J. Zhuang, G. Zöller, and Y. Ogata, “Forecasting the magnitude of the largest expected earthquake,” Nat. Commun. 10, 4051 (2019).
  * Lehnertz and Elger (1998) K. Lehnertz and C. E. Elger, “Can epileptic seizures be predicted? Evidence from nonlinear time series analysis of brain electrical activity,” Phys. Rev. Lett. 80, 5019–5023 (1998).
  * Streeter and Dugmore (2013) R. Streeter and A. J. Dugmore, “Anticipating land surface change,” Proc. Natl. Acad. Sci.. 110, 5779–5784 (2013).
  * Bialonski, Ansmann, and Kantz (2015) S. Bialonski, G. Ansmann, and H. Kantz, “Data-driven prediction and prevention of extreme events in a spatially extended excitable system,” Phys. Rev. E 92, 042910 (2015).
  * Grziwotz  _et al._ (2023) F. Grziwotz, C.-W. Chang, V. Dakos, E. H. van Nes, M. Schwarzländer, O. Kamps, M. Heßler, I. T. Tokuda, A. Telschow, and C.-h. Hsieh, “Anticipating the occurrence and type of critical transitions,” Sci. Adv. 9, eabq4558 (2023).
  * Benjamin  _et al._ (2018) D. J. Benjamin, J. O. Berger, M. Johannesson, B. A. Nosek, E.-J. Wagenmakers, R. Berk, K. A. Bollen, B. Brembs, L. Brown, C. Camerer, _et al._ , “Redefine statistical significance,” Nat. Hum. Behav. 2, 6–10 (2018).
  * Lehnertz, Rings, and Bröhl (2021) K. Lehnertz, T. Rings, and T. Bröhl, “Time in brain: How biological rhythms impact on EEG signals and on EEG-derived brain networks,” Front. Netw. Physiol. 1, 755016 (2021).
  * Dahlhaus and Neumann (2001) R. Dahlhaus and M. H. Neumann, “Locally adaptive fitting of semiparametric models to nonstationary time series,” Stoch. Process. Their Appl. 91, 277–308 (2001).
  * Carlstein (1988) E. Carlstein, “Nonparametric change-point estimation,” Ann. Stat. 16, 188–197 (1988).
  * Basseville, Nikiforov  _et al._ (1993) M. Basseville, I. V. Nikiforov, _et al._ ,  _Detection of abrupt changes: theory and application_ , Vol. 104 (Prentice Hall, Englewood Cliffs, NJ, USA, 1993).
  * Aminikhanghahi and Cook (2017) S. Aminikhanghahi and D. J. Cook, “A survey of methods for time series change point detection,” Knowl. Inf. Syst. 51, 339–367 (2017).
  * Cabrieto  _et al._ (2017) J. Cabrieto, F. Tuerlinckx, P. Kuppens, M. Grassmann, and E. Ceulemans, “Detecting correlation changes in multivariate time series: A comparison of four non-parametric change point detection methods,” Behav. Res. Methods 49, 988–1005 (2017).
  * Cabrieto  _et al._ (2018) J. Cabrieto, F. Tuerlinckx, P. Kuppens, B. Hunyadi, and E. Ceulemans, “Testing for the presence of correlation changes in a multivariate time series: A permutation based approach,” Sci. Rep. 8, 769 (2018).
  * Truong, Oudre, and Vayatis (2020) C. Truong, L. Oudre, and N. Vayatis, “Selective review of offline change point detection methods,” Signal Process. 167, 107299 (2020).
  * Bagniewski and Ghil (2021) W. Bagniewski and M. Ghil, “Automatic detection of abrupt transitions in paleoclimate records,” Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 113129 (2021).
  * Bury  _et al._ (2021) T. M. Bury, R. Sujith, I. Pavithran, M. Scheffer, T. M. Lenton, M. Anand, and C. T. Bauch, “Deep learning for early warning signals of tipping points,” Proc. Natl. Acad. Sci. 118, e2106140118 (2021).
  * Gilarranz  _et al._ (2022) L. J. Gilarranz, A. Narwani, D. Odermatt, R. Siber, and V. Dakos, “Regime shifts, trends, and variability of lake productivity at a global scale,” Proc. Natl. Acad. Sci. 119, e2116413119 (2022).
  * De Ryck, De Vos, and Bertrand (2021) T. De Ryck, M. De Vos, and A. Bertrand, “Change point detection in time series data using autoencoders with a time-invariant representation,” IEEE Trans. Sig. Proc. 69, 3513–3524 (2021).
  * Boettiger and Hastings (2012b) C. Boettiger and A. Hastings, “Quantifying limits to detection of early warning for critical transitions,” J. Roy. Soc. Interface 9, 2527–2539 (2012b).
  * Bunde, Ludescher, and Schellnhuber (2022) A. Bunde, J. Ludescher, and H. J. Schellnhuber, “How to determine the statistical significance of trends in seasonal records: application to Antarctic temperatures,” Clim. Dyn. 58, 1349–1361 (2022).
  * Daniel (2000) W. W. Daniel,  _Applied nonparametric statistics (revised ed.)_ (Duxbury, Pacific Grove, CA, USA, 2000).
  * Kolmogorov (1933) A. Kolmogorov, “Sulla determinazione empirica di una legge didistribuzione,” Giorn Dell’inst Ital Degli Att 4, 89–91 (1933).
  * Mann and Whitney (1947) H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” Ann. Math. Stat. 18, 50–60 (1947).
  * Yerushalmy (1947) J. Yerushalmy, “Statistical problems in assessing methods of medical diagnosis, with special reference to x-ray techniques,” Public Health Rep. (1896-1970) , 1432–1449 (1947).
  * Fawcett (2006) T. Fawcett, “An introduction to ROC analysis,” Pattern Recogn. Lett. 27, 861–874 (2006).
  * Egan (1975) J. P. Egan,  _Signal detection theory and ROC-analysis_ (Academic Press, Cambridge, MA, USA, 1975).
  * Drake (2013) J. M. Drake, “Early warning signals of stochastic switching,” Proc. Roy. Soc. B: Biological Sciences 280, 20130686 (2013).
  * Brinkmann  _et al._ (2016) B. H. Brinkmann, J. Wagenaar, D. Abbot, P. Adkins, S. C. Bosshard, M. Chen, Q. M. Tieng, J. He, F. J. Muñoz-Almaraz, P. Botella-Rocamora, J. Pardo, F. Zamora-Martinez, M. Hills, W. Wu, I. Korshunova, W. Cukierski, C. Vite, E. E. Patterson, B. Litt, and G. A. Worrell, “Crowdsourcing reproducible seizure forecasting in human and canine epilepsy,” Brain 139, 1713–1722 (2016).
  * Cavaliere  _et al._ (2016) M. Cavaliere, G. Yang, V. Danos, and V. Dakos, “Detecting the collapse of cooperation in evolving networks,” Sci. Rep. 6, 30845 (2016).
  * Kuhlmann  _et al._ (2018a) L. Kuhlmann, P. Karoly, D. R. Freestone, B. H. Brinkmann, A. Temko, A. Barachant, F. Li, G. Titericz Jr, B. W. Lang, D. Lavery, _et al._ , “Epilepsyecosystem. org: crowd-sourcing reproducible seizure prediction with long-term human intracranial EEG,” Brain 141, 2619–2630 (2018a).
  * Yang  _et al._ (2018) B. Yang, M. Li, W. Tang, W. Liu, S. Zhang, L. Chen, and J. Xia, “Dynamic network biomarker indicates pulmonary metastasis at the tipping point of hepatocellular carcinoma,” Nat. Commun. 9, 678 (2018).
  * Brett  _et al._ (2020) T. Brett, M. Ajelli, Q.-H. Liu, M. G. Krauland, J. J. Grefenstette, W. G. van Panhuis, A. Vespignani, J. M. Drake, and P. Rohani, “Detecting critical slowing down in high-dimensional epidemiological systems,” PLoS Comput. Biol. 16, e1007679 (2020).
  * Tredennick  _et al._ (2022) A. T. Tredennick, E. B. O’Dea, M. J. Ferrari, A. W. Park, P. Rohani, and J. M. Drake, “Anticipating infectious disease re-emergence and elimination: a test of early warning signals using empirically based models,” J. Roy. Soc. Interface 19, 20220123 (2022).
  * Dylewsky  _et al._ (2023) D. Dylewsky, T. M. Lenton, M. Scheffer, T. M. Bury, C. G. Fletcher, M. Anand, and C. T. Bauch, “Universal early warning signals of phase transitions in climate systems,” J. Roy. Soc. Interface 20, 20220562 (2023).
  * Gómez-Nava  _et al._ (2023) L. Gómez-Nava, R. T. Lange, P. P. Klamser, J. Lukas, L. Arias-Rodriguez, D. Bierbach, J. Krause, H. Sprekeler, and P. Romanczuk, “Fish shoals resemble a stochastic excitable system driven by environmental perturbations,” Nat. Phys. 19, 663–669 (2023).
  * Weiss (2004) G. M. Weiss, “Mining with rarity: a unifying framework,” ACM Sigkdd Explorations Newsletter 6, 7–19 (2004).
  * Ben Bouallègue and Richardson (2022) Z. Ben Bouallègue and D. S. Richardson, “On the ROC area of ensemble forecasts for rare events,” Weather and Forecasting 37, 787–796 (2022).
  * West  _et al._ (2023) J. West, Z. D. Bozorgi, J. Herron, H. J. Chizeck, J. D. Chambers, and L. Li, “Machine learning seizure prediction: one problematic but accepted practice,” J. Neural Eng. 20, 016008 (2023).
  * Mormann  _et al._ (2007) F. Mormann, R. Andrzejak, C. E. Elger, and K. Lehnertz, “Seizure prediction: the long and winding road,” Brain 130, 314–333 (2007).
  * Kuhlmann  _et al._ (2018b) L. Kuhlmann, K. Lehnertz, M. P. Richardson, B. Schelter, and H. P. Zaveri, “Seizure prediction – ready for a new era,” Nat. Rev. Neurol. 14, 618–630 (2018b).
  * Winterhalder  _et al._ (2003) M. Winterhalder, T. Maiwald, H. U. Voss, R. Aschenbrenner-Scheibe, J. Timmer, and A. Schulze-Bonhage, “The seizure prediction characteristic: A general framework to assess and compare seizure prediction methods,” Epilepsy Behav. 3, 318–325 (2003).
  * Schelter  _et al._ (2006) B. Schelter, M. Winterhalder, T. Maiwald, A. Brandt, A. Schad, A. Schulze-Bonhage, and J. Timmer, “Testing statistical significance of multivariate time series analysis techniques for epileptic seizure prediction,” Chaos 16, 013108 (2006).
  * Feldwisch-Drentrup  _et al._ (2011) H. Feldwisch-Drentrup, M. Staniek, A. Schulze-Bonhage, J. Timmer, H. Dickten, C. E. Elger, B. Schelter, and K. Lehnertz, “Identification of preseizure states in epilepsy: A data-driven approach for multichannel EEG recordings,” Front. Comput. Neurosci. 5, 32 (2011).
  * Mader  _et al._ (2014) M. Mader, W. Mader, B. J. Gluckman, J. Timmer, and B. Schelter, “Statistical evaluation of forecasts,” Phys. Rev. E 90, 022133 (2014).
  * Efron (1982) B. Efron,  _The Jackknife, the Bootstrap and Other Resampling Plans_ (Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 1982).
  * Efron and Tibshirani (1998) B. Efron and R. J. Tibshirani,  _An Introduction to the Bootstrap_ (Chapman & Hall/CRC, Boca Raton, London, New York, Washington DC, 1998) p. 436.
  * Schreiber and Schmitz (2000) T. Schreiber and A. Schmitz, “Surrogate time series,” Physica D 142, 346–382 (2000).
  * Lucio, Valdés, and Rodríguez (2012) J. Lucio, R. Valdés, and L. Rodríguez, “Improvements to surrogate data methods for nonstationary time series,” Phys. Rev. E 85, 056202 (2012).
  * Lancaster  _et al._ (2018) G. Lancaster, D. Iatsenko, A. Pidde, V. Ticcinelli, and A. Stefanovska, “Surrogate data for hypothesis testing of physical systems,” Phys. Rep. 748, 1–60 (2018).
  * Ansmann and Lehnertz (2011) G. Ansmann and K. Lehnertz, “Constrained randomization of weighted networks,” Phys. Rev. E 84, 026103 (2011).
  * Ansmann and Lehnertz (2012) G. Ansmann and K. Lehnertz, “Surrogate-assisted analysis of weighted functional brain networks,” J. Neurosci. Methods 208, 165–172 (2012).
  * Bialonski (2012) S. Bialonski, “Inferring complex networks from time series of dynamical systems: Pitfalls, misinterpretations, and possible solutions,” arXiv 1208.0800 (2012).
  * Laut and Räth (2016) I. Laut and C. Räth, “Surrogate-assisted network analysis of nonlinear time series,” Chaos 26, 103108 (2016).
  * Wiedermann  _et al._ (2016) M. Wiedermann, J. F. Donges, J. Kurths, and R. V. Donner, “Spatial network surrogates for disentangling complex system structure from spatial embedding of nodes,” Phys. Rev. E 93, 042308 (2016).
  * Stahn and Lehnertz (2017) K. Stahn and K. Lehnertz, “Surrogate-assisted identification of influences of network construction on evolving weighted functional networks,” Chaos 27, 123106 (2017).
  * Chorozoglou and Kugiumtzis (2019) D. Chorozoglou and D. Kugiumtzis, “Testing the randomness of correlation networks from multivariate time series,” J. Complex Netw. 7, 190–209 (2019).
  * Andrzejak  _et al._ (2003) R. G. Andrzejak, F. Mormann, T. Kreuz, C. Rieke, A. Kraskov, C. E. Elger, and K. Lehnertz, “Testing the null hypothesis of the nonexistence of a preseizure state,” Phys. Rev. E 67, 010901(R) (2003).
  * Kreuz  _et al._ (2004) T. Kreuz, R. G. Andrzejak, F. Mormann, A. Kraskov, H. Stögbauer, C. E. Elger, K. Lehnertz, and P. Grassberger, “Measure profile surrogates: A method to validate the performance of epileptic seizure prediction algorithms,” Phys. Rev. E 69, 061915 (2004).
  * Bertsimas and Tsitsiklis (1993) D. Bertsimas and J. Tsitsiklis, “Simulated annealing,” Stat. Sci. 8, 10–15 (1993).
  * Kramer  _et al._ (2012) M. A. Kramer, W. Truccolo, U. T. Eden, K. Q. Lepage, L. R. Hochberg, E. N. Eskandar, J. R. Madsen, J. W. Lee, A. Maheshwari, E. Halgren, C. J. Chu, and S. S. Cash, “Human seizures self-terminate across spatial scales via a critical transition,” Proc. Natl. Acad. Sci. U.S.A. 109, 21116–21121 (2012).
  * Boettner and Boers (2022) C. Boettner and N. Boers, “Critical slowing down in dynamical systems driven by nonstationary correlated noise,” Phys. Rev. Res. 4, 013230 (2022).
  * Chen, Ghadami, and Epureanu (2022) S. Chen, A. Ghadami, and B. I. Epureanu, “Practical guide to using Kendall’s $\tau$ in the context of forecasting critical transitions,” Roy. Soc. Open Sci. 9, 211346 (2022).
  * Pal, Deb, and Dutta (2022) K. Pal, S. Deb, and P. S. Dutta, “Tipping points in spatial ecosystems driven by short-range correlated noise,” Phys. Rev. E 106, 054412 (2022).
  * Bochow and Boers (2023) N. Bochow and N. Boers, “The South American monsoon approaches a critical transition in response to deforestation,” Sci. Adv. 9, eadd9973 (2023).
  * Shmueli (2010) G. Shmueli, “To explain or to predict?” Stat. Sci. 25, 289–310 (2010).
  * Cook  _et al._ (2013) M. J. Cook, T. J. O’Brien, S. F. Berkovic, M. Murphy, A. Morokoff, G. Fabinyi, W. D’Souza, R. Yerra, J. Archer, L. Litewka, S. Hosking, P. Lightfoot, V. Ruedebusch, W. D. Sheffield, D. Snyder, K. Leyde, and D. Himes, “Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study,” Lancet Neurol. 12, 563–571 (2013).
  * Ludescher, Bunde, and Schellnhuber (2023) J. Ludescher, A. Bunde, and H. J. Schellnhuber, “Forecasting the El Niño type well before the spring predictability barrier,” npj Clim. Atmos. Sci. 6, 196 (2023).
  * Gneiting and Katzfuss (2014) T. Gneiting and M. Katzfuss, “Probabilistic forecasting,” Annu. Rev. Stat. Appl. 1, 125–151 (2014).
  * Tashman (2000) L. J. Tashman, “Out-of-sample tests of forecasting accuracy: an analysis and review,” Int. J. Forecast. 16, 437–450 (2000).
  * Gneiting, Balabdaoui, and Raftery (2007) T. Gneiting, F. Balabdaoui, and A. E. Raftery, “Probabilistic forecasts, calibration and sharpness,” J. R. Stat. Soc., B: Stat. Methodol. 69, 243–268 (2007).
  * Gneiting (2011) T. Gneiting, “Making and evaluating point forecasts,” J. Am. Stat. Assoc. 106, 746–762 (2011).
  * Lerch  _et al._ (2017) S. Lerch, T. L. Thorarinsdottir, F. Ravazzolo, and T. Gneiting, “Forecaster’s dilemma: extreme events and forecast evaluation,” Statist. Sci. 32, 106–127 (2017).
  * Gneiting  _et al._ (2023) T. Gneiting, D. Wolffram, J. Resin, K. Kraus, J. Bracher, T. Dimitriadis, V. Hagenmeyer, A. I. Jordan, S. Lerch, K. Phipps, _et al._ , “Model diagnostics and forecast evaluation for quantiles,” Annu. Rev. Stat. Appl. 10, 597–621 (2023).
  * Berger and Smith (2019) J. O. Berger and L. A. Smith, “On the statistical formalism of uncertainty quantification,” Annu. Rev. Stat. Appl. 6, 433–460 (2019).