# The Role of Temporal Hierarchy in Spiking Neural Networks

Filippo Moro Corresponding author<EMAIL_ADDRESS>Institute of
Neuroinformatics, UZH and ETH Zurich, Zurich, CH  Pau Vilimelis Aceituno
Institute of Neuroinformatics, UZH and ETH Zurich, Zurich, CH  Laura Kriener
Institute of Neuroinformatics, UZH and ETH Zurich, Zurich, CH  Melika Payvand
Institute of Neuroinformatics, UZH and ETH Zurich, Zurich, CH

###### Abstract

Spiking Neural Networks have the potential for rich spatio-temporal signal
processing thanks to exploiting both spatial and temporal parameters. The
temporal dynamics such as time constants of the synapses and neurons and
delays have been recently shown to have computational benefits that help
reduce the overall number of parameters required in the network and increase
the accuracy of the SNNs in solving temporal tasks. Optimizing such temporal
parameters, for example, through gradient descent, gives rise to a temporal
architecture for different problems. As has been shown in machine learning, to
reduce the cost of optimization, architectural biases can be applied, in this
case in the temporal domain. Such inductive biases in temporal parameters have
been found in neuroscience studies, highlighting a hierarchy of temporal
structure and input representation in different layers of the cortex.
Motivated by this, we propose to impose a hierarchy of temporal representation
in the hidden layers of SNNs, highlighting that such an inductive bias
improves their performance. We demonstrate the positive effects of temporal
hierarchy in the time constants of feed-forward SNNs applied to temporal tasks
(Multi-Time-Scale XOR and Keyword Spotting, with a benefit of up to 4.1% in
classification accuracy). Moreover, we show that such architectural biases,
i.e. hierarchy of time constants, naturally emerge when optimizing the time
constants through gradient descent, initialized as homogeneous values. We
further pursue this proposal in temporal convolutional SNNs, by introducing
the hierarchical bias in the size and dilation of temporal kernels, giving
rise to competitive results in popular temporal spike-based datasets.

_Keywords_ Spiking Neural Networks $\cdot$ Hierarchy of Time Constants $\cdot$
Temporal Architecture

##  1 Introduction

SNNs utilize time as an additional dimension for computation and have thus
been shown promising for temporal data processing [1]. As their computational
unit, SNNs incorporate spiking neurons that integrate information over time in
their cell body and generate an electric pulse, i.e., a spike, when this value
passes a threshold. Arguably, the spikes convey this information through the
timing of their generation, or their frequency [2]. Different models of
spiking neurons have been developed, from simplified Leaky Integrate and Fire
(LIF) models to more biologically realistic ones [3]. To perform computation,
SNNs exploit the dynamics of such spiking neurons for generating short- and
long-term memory [4].

Figure 1: Temporal hierarchy in Spiking Neural Networks (SNNs). a) Multi-Layer
SNN featuring $\mathcal{N}$ hidden layers, each colored differently,
highlighting the difference in temporal processing speed of the neurons in the
hidden layers, from Fast in the first hidden layers to Slow in the deeper
layers. b) Time constant $\mathbf{\tau}$ determining the speed of neuronal
dynamics in Leaky-Integrate-and-Fire (LIF) spiking neurons. The differential
equation describes the dynamics for the membrane voltage $u(t)$, which evolves
with a time constant $\mathbf{\tau}$. c) Temporal processing expressed in
temporal convolutions by varying the weights’ kernel $W_{k}$, affecting the
state of neurons $u(t)$ in time.

It has been shown that optimizing these dynamics in SNNs increases the
accuracy and/or reduces the number of parameters of the SNNs in temporal
tasks. These optimized dynamics include the leaky dynamics of the neurons
(i.e. their time constant) [5], delay parameters in the synapses [6, 7] or
temporal causal convolutions [8], adaptation of the threshold voltage of the
neurons [4, 9, 10], and recurrent connections resulting in short-term memory
[11, 12].

While these computational principles are commonly featured in SNNs, reaching
their full potential remains an open research problem. In this work, we take a
step in combining understandings from the machine learning and computational
neuroscience literature to further exploit the temporal dynamics of SNN for
better performance.

On the one hand, the success of Deep Neural Networks (DNNs) has shown the
computational benefits of hierarchical representation of inputs in successive
deep layers [13, 14], and recently, such hierarchical input representation has
also been applied to temporal data [15].

On the other hand, recent neuroscience findings highlight the beneficial role
of heterogeneity introduced to the dynamics of SNNs. For example, it has been
shown that exploiting a variety of time constants in a population of neurons
improves the performance of SNNs [16, 17].

Moreover, the concept of the hierarchy in structure [18, 19], input and action
representation [20, 21] in the brain is well studied in neuroscience. Authors
in [22] show that the hierarchical organization in the cortex is not only used
for spatial operations but also for temporal operations. They also prove that
early sensory regions tend to be affected by more recent input features, while
the activity of higher-order cortical regions correlates with longer windows
of prior stimulus context. The timescales of fluctuation in spiking activity
have been observed to increase across the hierarchy of cortical areas in
primates [23]. This evidence hints that temporal processing is organized in a
hierarchy in the brain, and this might be crucial to developing complex
behavior.

Combining these lessons from DNNs and computational neuroscience, i.e.
benefits of hierarchical representation and heterogeneity, we analyze the
implications of imposing a hierarchy of temporal processing speed in SNNs. We
propose to leverage two important temporal processing elements of SNNs, neural
dynamics in the form of the time constant and delays or temporal causal
convolutions. With these elements, we construct a hierarchy of temporal
representation in multi-layer SNNs (Figure 1a) that brings a performance
benefit in common spike-based benchmark tasks. Also, we show that performance
decreases if the hierarchy of the temporal representation is reversed. In
general, we advocate for a simple setting in initializing SNNs, whereby
temporal processing elements are ordered from fast to slow through the hidden
layers of the SNN.

Different sets of experiments are performed to prove the hypothesis on the
benefit of temporal hierarchy, testing SNNs on common spike-based benchmark
tasks, such as the Multi-Time-Scale XOR [17], the Spiking Heidelberg Digit
(SHD) [24] and the spike-based version of the Google Speech Command dataset
(SSC) [25, 24]. First, the time constants of neurons are initialized in a
crescendo through the hidden layers of SNNs. This proves to be a useful
inductive bias in forming a complex representation of the input and improving
its computational power of the SNNs, with a performance increase of up to 4.1%
in the Spiking Heidelberg Digits dataset. Additionally, we find that such
temporal hierarchy naturally emerges through optimization, as a result of
training the time constants of the neurons of an SNN with backpropagation. We
also demonstrate that the introduction of temporal hierarchy is useful in SNNs
based on temporal causal convolutions, where the kernel and dilation of the
convolutions are arranged hierarchically in the network, reaching very
competitive classification accuracy in keyword-spotting datasets, up to 94.1%
on the SHD and 79.2% on the SSC.
Overall, our results shed light on the role of the temporal hierarchy as a
positive inductive bias for SNNs and provide a simple heuristic to improve the
performance of SNNs in temporal tasks.

##  2 Results

To verify the hypothesis that a hierarchy of temporal processing is beneficial
to SNNs, we set up a series of experiments involving common benchmark tasks.
In particular, the benefit is expected when the SNN processes data with a
temporal structure. To prove this, we choose three relevant temporal tasks:
the multi-time-scale (MTS) XOR [17], the Spiking Heidelberg Digits (Keyword
Spotting) [24], and the Spiking Speech Command (Keyword Spotting) [24, 25].
Additionally, we investigate the role of temporal hierarchy with static data
such as images (see Supplementary Figure S1a). As temporal features do not
play a role in such a task, temporal hierarchy is expected not to provide
added benefits. However, it is possible to revert static images to sequences,
by feeding SNNs single pixels at a time. In this case, temporal features
emerge and temporal hierarchy should boost SNNs’ performance.

Temporal processing can be expressed in multiple forms in SNNs. Neuronal
dynamics is the most characteristic one in SNNs. Lately, delays (or causal
temporal convolutions) emerged as a promising temporal processing element,
allowing SNNs to improve performance considerably [7, 8, 6]. As such, we
express the concept of temporal hierarchy in two main ways: first, as the
difference of the time constant in the hidden layers of SNNs (Figure 1b);
second, as the varying kernel size and dilation of causal temporal
convolutions in SNNs (Figure 1c). In both cases, we aim to show that arranging
the temporal computational features from fast to slow, through the hidden
layers of SNNs, is beneficial for performance. The experiments are organized
in three sets, each providing intuition on the role of temporal hierarchy in
SNNs:

  1. (i)

Hierarchy of time constant in feed-forward SNNs

  2. (ii)

Discovering hierarchy via the optimization of time constants

  3. (iii)

Temporal hierarchy in convolutional SNNs

Along with the experiments, we compare the benefits of temporal hierarchy with
other inductive biases in SNNs, confronting our results with the state of the
art.

Figure 2: Temporal hierarchy through the time constants of SNNs. a) Sketch of
the Multi-Time-Scale XOR (MTS-XOR) task. b) Classification accuracy on MST-XOR
as a function of the homogeneous time constant through the hidden layers. c)
Classification accuracy when temporal hierarchy is introduced in multi-layer
SNNs with varying amplitude ($\color[rgb]{0,0.88,0}\Delta\tau$). Positive
$\color[rgb]{0,0.88,0}\Delta\tau$ improves the classification accuracy. d)
Sample from the Spiking Heidelberg Digits (SHD) dataset. e) The time constant
hierarchy is parametrized according to a tanh function, with steepness
($\color[rgb]{0.55,0.14,1}s$) and centering ($\color[rgb]{0,0,1}c$)
parameters. f) The effect of such parameters is tested on the SHD task, with
${\color[rgb]{0,0.88,0}\Delta\tau}=$+150\text{\,}\mathrm{ms}$$. In all cases,
hierarchy has a positive effect compared to the reference performance
(${\color[rgb]{0,0.88,0}\Delta\tau}=$0\text{\,}\mathrm{ms}$$). g) Temporal
hierarchy is applied to networks of varying size, showing that an SNN with 32
hidden neurons per layer and temporal hierarchy performs as well as a bigger
SNN with 128 neurons per layer without temporal hierarchy. Error bars show the
quartiles over 10 trials in all plots.

###  2.1 Time constant hierarchy in Feed-Forward SNN

Neuronal dynamics is the most characteristic temporal processing feature in
SNNs; the spectrum of possible neuron models ranges from more bio-realist
multi-compartment implementations [26], to simplified point-like structures
e.g., used in McCulloch-Pitts. The LIF model has been widely adopted in the
neuromorphic community for its trade-off between expressivity of neuronal
dynamics, computational complexity, and ease of hardware implementation. The
LIF model features a state variable - often called membrane voltage $u(t)$ \-
which evolves in time according to a first-order differential equation (Figure
1b). The temporal processing of LIF neurons is expressed by the time constant
$\tau$ of the membrane voltage, which enables them to detect the coincidence
between two input spikes, aka Coincidence Detection [27, 28], and to filter
input spike patterns based on their precise timing [29]. In this analysis, we
examine the effect of implementing a temporal hierarchy via time constants in
feed-forward SNNs. The time constants of LIF neurons are set to be fast in the
initial layers and slow in the deeper layers. We use a feed-forward network to
isolate the impact of the time-constant hierarchy. While recurrent networks
can control neuron “speed” and memory by training recurrent weights, we
demonstrate that the resulting working memory can be achieved through the
neuron’s inherent dynamics alone. Here, the time constants are not trained
after initialization; this aspect will be addressed in section 2.2.

#### MTS-XOR task

We train feed-forward SNNs with 2 and 3 hidden layers on the MTS-XOR dataset
[17], each with 10 neurons per hidden layer. The MTS-XOR (Fig. 2a) is a
synthetic dataset where a single cue is provided at $t=$0\text{\,}\mathrm{s}$$
from one of two channels (Channel A) - composed of a population of neurons -
with multiple other cues successively provided from the other channel (Channel
B) in time. Cues are binary activations of $20\text{\,}\mathrm{ms}$ each
expressed as Poisson firing rates, and the task consists of performing the XOR
operation each time a cue is generated from Channel B, based on the original
cue in Channel A, which is not repeated. The total duration of each datapoint
is $1\text{\,}\mathrm{s}$ and the time-step is set to
$10\text{\,}\mathrm{ms}$. Each cue is presented for $100\text{\,}\mathrm{ms}$
with a Poisson frequency of either $10\text{\,}\mathrm{Hz}$ (activation 1) or
$10\text{\,}\mathrm{kHz}$ (activation 0). A break of $50\text{\,}\mathrm{ms}$
is interposed between cues. The task can be successively solved if the network
holds the information of the first cue while being able to react quickly to
the new cues and perform the XOR operation. In our preliminary analysis, we
set the time constants in a 2- or 3-hidden layer SNN to a fixed value,
$\tau_{\mu}$, which we varied to identify the optimal value for the MTS-XOR
task. Figure 2b shows that a time constant of $300\text{\,}\mathrm{ms}$ is
sufficient to achieve maximum classification accuracy. Fixing the average time
constant to $\tau_{\mu}=$300\text{\,}\mathrm{ms}$$, we then modify the time
constant of hidden layers $\tau(l)$ to create a hierarchy of time constant
proportional to $\color[rgb]{0,0.88,0}\Delta\tau$.

$\tau(l)=\tau_{\mu}+(l-H_{l}/2){\color[rgb]{0,0.88,0}\Delta\tau}/2$

We then train SNNs with different values of $\color[rgb]{0,0.88,0}\Delta\tau$
in Fig.2c. Following the hypothesis on temporal hierarchy, we show that
classification accuracy positively correlates with the hierarchy amplitude
$\color[rgb]{0,0.88,0}\Delta\tau$, i.e. the time constant difference across
the hidden layers of the feed-forward SNN. In an SNN with 2 hidden layers,
having a fast first layer and a slow last layer is beneficial. Adding a third
hidden layer generally improves performance and makes the impact of temporal
hierarchy less prominent, yet the correlation of performance with the
hierarchy amplitude persists. At the same time, reversing the order of the
hierarchy reduces the computational power of the SNNs. We hypothesize that
this could be due to the functional role of different layers in a multi-layer
SNN, with the first hidden layer acting as a feature extractor, while the
deeper hidden layers integrate information over longer time spans. To verify
this, we performed additional experiments in the Supplementary Information
Feed Forward SNN with temporal hierarchy. As the MTS-XOR dataset includes
background noise, this hypothesis indicates that having a first fast hidden
layer would help filter out such noise, while the information of the first cue
is “stored” in the slow dynamics of the second layer. If the first hidden
layer was slower, the noise would be integrated and the memory of the first
cue would be lost. This is confirmed in the Supplementary Figure S1a, where
the classification accuracy in the MTS-XOR task is analyzed as a function of
the background noise rate. Positive temporal hierarchy (fast to slow dynamics)
makes the SNN more resilient to noise.

#### Keyword Spotting

In the second set of experiments temporal hierarchy is tested on the SHD task
(Fig. 2d), a common benchmark in the spike-based processing domain, consisting
of audio recordings of spoken digits in two languages (English and German)
converted to trains of spikes. Each spoken digit is recorded for about
$1\text{\,}\mathrm{s}$ and at $16\text{\,}\mathrm{kHz}$ and converted to the
spike domain with a bio-plausible cochlear model [24].

Given that SHD is a more complex task than MTS-XOR, we utilize a deeper SNN
with up to 5 hidden layers, with 32 neurons per hidden layer. With more layers
available, the time constant for each layer is determined with a hyperbolic
tangent $\tanh$ function, to explore the optimal “shape” of temporal
hierarchy.

$\tau(l)=\tau_{\mu}+\tanh_{\mathrm{norm}}\left({\color[rgb]{0.55,0.14,1}s}(l/H_{l}-{\color[rgb]{0,0,1}c}\right)){\color[rgb]{0,0.88,0}\Delta\tau}/2$

Hierarchy is thus parameterized by the steepness
($\color[rgb]{0.55,0.14,1}{s}$), centering ($\color[rgb]{0,0,1}{c}$), and
amplitude ($\color[rgb]{0,0.88,0}{\Delta\tau}$) factors. The resulting shapes
of the time constant hierarchy are shown in Figure 2e. The effect of the
steepness and centering parameters are analyzed while fixing the hierarchy
amplitude at ${\color[rgb]{0,0.88,0}{\Delta\tau}}=$150\text{\,}\mathrm{ms}$$
in Figure 2e, where 5-hidden layers feed-forward SNNs are trained on the SHD
task. The average time constant $\tau_{\mu}$ is set to
$200\text{\,}\mathrm{ms}$ as it maximizes the performance (see Supplementary
Figure S1b). We show in a color map (Figure 2f) the classification accuracy
with temporal hierarchy, compared to the case
${\color[rgb]{0,0.88,0}{\Delta\tau}}=$0\text{\,}\mathrm{s}$$. Notably, all
values are positive, confirming that temporal hierarchy correlates with a
boost in performance. Certain combinations of steepness and centering
parameters provide a larger increase in classification accuracy, with a peak
of 4.1% when ${\color[rgb]{0.55,0.14,1}{s}}=0.5$ and
${\color[rgb]{0,0,1}{c}}=0.5$. The tanh shapes that yield the best performance
are illustrated in Supplementary Figure S1d. Fixing this combination of
optimal hierarchy parameters, the number of neurons per hidden layer is
increased to 128 neurons, and the results are compared with and without
temporal hierarchy for increasingly deeper networks (Figure 2f). Temporal
hierarchy improves performance for both shallow and deep SNNs. Interestingly,
the SNNs with 32 neurons per hidden layer with temporal hierarchy perform on
par with larger SNNs with 128 neurons per hidden layer and homogeneous time
constants. This is a considerable achievement, as the latter network has 6
times more parameters. From a hardware-centered perspective, temporal
hierarchy allows shrinking the size of an SNN model while granting the same
level of performance. For an estimation of the effect of the hierarchy
amplitude $\color[rgb]{0,0.88,0}{\Delta\tau}$ on the performance, refer to
Supplementary Figure S1c.
To sum up, we report the results on the SHD dataset with the temporal
hierarchy-endowed SNNs (hSNNs) in Table 1. Furthermore, the hierarchy of time
constants is tested on the Spiking Speech Command (SSC) dataset [25], showing
a performance boost compared with a network with homogeneous time constants.
These results are found in Table 2.

###  2.2 Optimization discovers time constant hierarchy

Figure 3: Optimizing the time constants in SNNs. a) Multi-layer SNNs are
initialized with the same distribution of time constants through the hidden
layers. Each time constant is optimized individually. b) Probability Density
Function for the optimized time constants in a 3-hidden-layer SNN solving the
MTS-XOR task. The median time constant grows in each layer, indicating the
formation of a hierarchy. c) Mean time constant from 5-hidden-layer SNNs
solving the SHD task. The mean grows indicating the hierarchy through the
network. Results are averaged from 5 trials in both b) and c).

The experiments in the previous section prove the efficacy of temporal
hierarchy in the initialization of time constants of LIF neurons as an
inductive bias to SNNs. However, time constants were not optimized in these
experiments. Here we observe the results of optimizing the time constants of
LIF neurons in SNNs, looking for a pattern in these trained values. To find
out whether a hierarchy of time constants naturally emerges from optimization,
we initialize SNNs with a neuron time constant drawn from a Gaussian
distribution with a given mean ($\tau_{\mu}$) and standard deviation of 20% of
the mean. Such initialization is the same for all the hidden layers in the
SNNs. Following a conventional gradient descent approach, we update the time
constant of hidden neurons while training the SNNs to solve the MTS-XOR and
SHD tasks (Fig 3a). For more details on the optimization of time constants,
refer to Method section 4.2. The probability density function (PDF) of the
time constant in the hidden layer of 5 SNNs trained on the MTS-XOR is shown in
Figure 3b. As previously observed in [16], the distribution of optimized time
constants (PDFs in green) in SNNs follows fat-tailed log-normal distributions.
Along with the PDF, we indicate the median of the distributions in purple.
Notably, the median values grow from layer to layer in the SNN. While the
probability density function (PDF) appears similar in the first and second
hidden layers, the PDF in the third layer features a larger tail, causing the
median value to be larger. This confirms that the hierarchy of time constants
is generated from optimization.
As a further confirmation, we repeated the experiment with 5-hidden layer SNNs
trained on the SHD task. Again, the methodology follows what is described in
Fig 3a, and we show the resulting mean values of the time constant
distribution per hidden layer. Once again, the mean value grows through the
layers indicating that the optimization results in a temporal hierarchy formed
by the time constants.

###  2.3 Temporal hierarchy in convolutional SNN

Figure 4: Temporal hierarchy in convolutional Spiking Neural Networks. a)
Kernel size can vary in 1D-Causal convolution. One can increase the kernel
size across hidden layers, forming a hierarchy of temporal representation. b)
Effect of the hierarchy of kernel size on the SHD task. Positive hierarchy
increases the classification accuracy. c) Dilation in Causal Temporal
Convolution controls the size of the receptive field. One can construct a
temporal hierarchy by progressively enlarging the dilation across the hidden
layers. d) Effect of the hierarchy of dilations on the SHD task. Incrementing
dilation through the hidden layers improves the performance of the network.
Error bars show the quartiles over 10 trials.

Delays and Temporal Causal Convolutions have been demonstrated extremely
effective in improving the expressiveness of SNNs [6, 8, 30, 31, 7, 32]. In
particular, DenRAM [6] introduces synaptic delays in dendritic-like branches
that can be modeled as temporal convolutions with a kernel size of 1 and
variable - though fixed after initialization - dilation, showing superior
performance with minimized hardware resources and parameter count. These
results are extended in [8] where multi-layer SNNs are equipped with temporal
causal convolutions whose dilation can be learned. These works either assume
shallow SNNs [6] or do not differentiate between the convolutions in different
layers [8] with specific initialization or training policies. Here, we perform
experiments with kernel size and dilation in temporal convolutions in SNNs,
demonstrating that imposing temporal hierarchy can yield better performance.
Preliminary results on kernel size and dilation hyper-parameter optimization
are shown in Supplementary Figure S3.
The kernel size in a temporal convolution corresponds to the size of the
convolutional window, i.e. the number of delay parameters used to compute a
given neuron’s activation, as illustrated in Figure 4a. We construct a
temporal hierarchy by increasing the kernel size at every layer, denoting as
$\Delta_{\mathrm{ker}}$, the kernel size difference across the network, i.e.
the amplitude of the temporal hierarchy. Based on a network of two hidden
layers, with 128 neurons per hidden layer, dilation fixed at five, trained to
solve the SHD task, we analyze the effect of temporal hierarchy constructed
with the kernel size in Figure 4b. While the mean kernel size is five, we
sweep $\Delta_{\mathrm{ker}}$ between $-4$ and $+4$, noting that performance
positively correlates with the amplitude of temporal hierarchy. This confirms
the trend observed through the experiments in Fig. 2.
The dilation in temporal convolution determines the size of the receptive
fields of post-synaptic neurons, i.e. the extent of the delays to compute the
neuron’s activation, as illustrated in Figure 4c. Temporal hierarchy is here
constructed by progressively increasing dilation through the layers of the
SNN. We construct a temporal hierarchy by increasing the dilation at every
layer, denoting the dilation size difference across the network as
$\Delta_{\mathrm{dil}}$. The effect of such parameters on classification
performance on the SHD datasets is shown in Figure 4d demonstrates a positive
correlation between classification accuracy and temporal hierarchy. This
suggests once again that incorporating temporal hierarchy may serve as an
effective inductive bias for improving SNN performance. The kernel size and
dilation hierarchy are then combined, along with data augmentation, to
maximize the performance of SNNs. The results are summarized in Table 1, where
we show how temporal hierarchy allows SNNs to reach competitive performance
compared with the state-of-the-art. The best performing convolutional SNN
features $\Delta_{\mathrm{ker}}=3$ and $\Delta_{\mathrm{dil}}=2$.
Note that other factors play a crucial role in significantly enhancing
performance. The type of data augmentation used in the experiments is detailed
in the Methods section. Specifically, the “Test as Valid” column in Table 1
merits further discussion. The SHD dataset is originally split into a training
and test set, without a validation set. Common practice is to split the
training set into two sub-sets, with a conventional 80%-20% split, forming a
validation set. This method is utilized in [6, 33] and we also recommend this
approach. However, multiple publications feature another validation strategy,
where the test set is assumed as the validation set, and performance (along
with hyper-parameter optimization and validation early-stopping) is evaluated
on the test set. As this method leaks information from the test set into the
training phase, the generalization capabilities of the model can no longer be
evaluated properly. With the sole scope of comparing performance with works
utilizing such validation scheme [8, 34], we also report results obtained with
this method (“Test as Valid”). A thorough comparison with the state-of-the-art
in SHD is reported in the Supplementary Information (Table S2).
Additionally, we performed experiments on the SSC dataset with temporal
convolution hierarchy, once again observing a slight increase in performance.
Results are reported in Table 2.

| Hierarchy |  | Data
---
Augment
| Test as
---
Valid*
| # hid.
---
layers
| hidden
---
size
# params | Accuracy%
hSNN |  |  |  | 3 | 32 | 25k | 81.2 ± 1.4
✓ |  |  | 3 | 32 | 25k | 85.6 ± 1.2
|  |  | 4 | 128 | 150k | 86.4 ± 1.3
✓ |  |  | 4 | 128 | 150k | 88.1 ± 0.6
hCSNN |  |  |  | 4 | 128 | 345k | 91.2 ± 0.8
✓ |  |  | 92.2 ± 1.1
✓ | ✓ |  | 94.1 ± 0.7
✓ | ✓ | ✓ | 94.7 ± 0.7

Table 1: Performance with and without temporal hierarchy of SNNs on the
Spiking Heidelberg Digits dataset.

| Hierarchy |  | Data
---
Augment
| # hid.
---
layers
| hidden
---
size
# params | Accuracy%
hSNN |  |  | 2 | 128 | 150k | 71.0 ± 1
✓ |  | 73.1 ± 0.3
hCSNN |  | ✓ | 2 | 256 | 1M | 78.2 ± 0.1
✓ | ✓ | 79.2 ± 0.2

Table 2: Performance with and without temporal hierarchy of SNNs on the
Spiking Speech Command dataset.

##  3 Discussion

This work proposes to organize temporal computational elements in a hierarchy,
from fast to slow, through the layers of SNNs. Temporal hierarchy is
constructed with two different computational elements: neuronal dynamics, i.e.
time constants in LIF neurons, and delays, i.e. temporal causal convolutions.
First, we show that organizing time constants in a hierarchy yields a
performance increment in common temporal benchmark tasks. Then we let
optimization learn the time constants in SNNs, finding that hierarchy emerges
from the process. The benefits of temporal hierarchy are also found in
temporal causal convolutional SNNs, where hierarchy is constructed from the
kernel size and dilation of the convolution. In all these cases, we benchmark
SNNs on temporal tasks, where the evolution of the data over time encodes
relevant information.

As a counter-example, we investigate the case where the input data is static,
i.e. it does not change over time, as in the case of images. In Supplementary
Figure S4 we classify MNIST digits with SNNs, showing that temporal hierarchy
plays no role in this case. This sets a boundary to the practical utility of
temporal hierarchy, which is beneficial only if the data features relevant
temporal information. As further proof, we change the encoding of MNIST
images, this time in the common “sequential MNIST” (sMNIST [35]) setting: the
information from each pixel is passed to an SNN sequential. In this setting,
the sequence encodes relevant information, and temporal hierarchy becomes
beneficial again, providing a boost of classification accuracy.

#### Performance benefit

In the experiments on temporal hierarchy constructed with time constants
(Table 1), the benefit in the performance of the SHD task was 4.1% with a
small size SNN (32 neurons per hidden layer) and it reduced to 1.5% with a
larger SNN (128 neurons per hidden layer). When constructing temporal
hierarchy with convolutions the benefit of hierarchy on the SHD task further
decreased to 1% (Table 1). We also show how data augmentation can yield a
comparable performance boost, raising the question of the true power of
temporal hierarchy. While we acknowledge that the performance benefit is not
drastic, the power of temporal hierarchy is in its simplicity and generality.
It is simple to implement, as it acts as a rule for initialization. It is
general as it has been proven beneficial for the most common computational
primitives in SNNs, neuronal dynamics and delays. Furthermore, the experiments
run in this work involve datasets with limited complexity. Possibly, testing
on datasets with more complex temporal features could reveal even greater
benefits in terms of performance.

#### Implications for neuromorphic computing

The hardware implementation of SNNs is the main aim of neuromorphic
engineering. Many existing neuromorphic chips [36, 37, 38] support the LIF
neuron model, and some circuits to implement dendritic delays have been
developed [6, 39, 40]. This makes temporal hierarchy a valuable feature that
most neuromorphic hardware designers can easily leverage. For example, in
Figure 2g, we show that temporal hierarchy allows reducing the size of an SNN
by $6\times$, while maintaining the same performance on the SHD task. In this
manner, the energy consumption and memory footprint of a neuromorphic chip
running an SNN can be considerably reduced. Furthermore, temporal hierarchy
has been demonstrated on multiple temporal processing primitives. This
indicates that temporal hierarchy can be customized to the specifics of each
neuromorphic chip implementation, providing a performance benefit independent
of the chip’s design choices.

#### Outlook

In addition to our demonstration of the benefit of temporal hierarchy in
multiple settings, there are numerous ways to extend this work. On the one
hand, more complex neuronal dynamics can be leveraged to form temporal
hierarchies. The true potential of the dynamics of, for example, Izhikevich
neurons [41] or multi-compartment models could be unlocked by a hierarchical
setting of their hyper-parameters. For example, one could build a temporal
hierarchy exploiting complex computational primitives of biologically
plausible neuron models, such as plateau potentials, spike bursts, and
intrinsic plasticity. On the other hand, the benefit of temporal hierarchy
could be even greater when tested on datasets with a higher degree of temporal
information. In some cases, temporal hierarchy is a natural feature of data.
In speech, information is represented across multiple levels: phonemes, words,
phrases, and semantics. A single model that tackles all these input
representations will likely benefit from a temporal hierarchy.

##  4 Methods

###  4.1 SNN implementation

For all our experiments, we implement SNNs using the JAX [42] python library
to benefit from auto-differentiation and speed of execution. We run the
experiments on two A6000 GPUs. Spiking Neural Networks are formalized based on
the Leaky-Integrate-and-Fire equation:

$\displaystyle\tau\dot{u}(t)$
$\displaystyle=-u(t)+f(I(t),W)-u_{\mathrm{th}}s(t)$ $\displaystyle s(t)$
$\displaystyle=\theta(u(t)-u_{\mathrm{th}})$

where, $u(t)\in\mathcal{R}^{N}$ is the membrane voltage of the neuron,
$\tau\in\mathcal{R}^{N}$ its time constant,
$W\in\mathcal{R}^{M,N}~{}or~{}\mathcal{R}^{K,M,N}$ the input weights and
$I\in\mathcal{R}^{M}$ the layer’s inputs. $N$ is number of neurons, $M$ number
of input connections, $K$ kernel size (when in convolution mode). The neuron
emits a spike ($s(t)\in\mathcal{R}^{N}$) whenever the membrane voltage $u(t)$
crosses the threshold $u_{\mathrm{th}}$. We set $u_{\mathrm{th}}=1$ for all
our experiments. After a spike, the membrane voltage is reset: we chose a
“soft reset” mechanism, i.e. the membrane voltage is not immediately reset to
zero, but it is decreased proportionally to the threshold voltage.
As it is well known, the non-linearity of a LIF neuron (Heaviside function) is
non-differentiable. To allow the gradient to flow, we use the Surrogate
Gradient technique [43], replacing the derivative of the Heaviside function
with a “box function”, as in [9].
Based on the equations above, we build layers of spiking neurons connected by
either fully connected layers

$f(I(t),W)=I(t)\,W\quad\text{with}\quad W\in\mathcal{R}^{M,N}$

or by 1D temporal causal convolution

$f(I(t),W)=I(t)*W\quad\text{with}\quad W\in\mathcal{R}^{K,M,N}$

In the latter case, we pad the input sequence $I(t)$ with as many zeros as
$K-1$ to make the temporal convolution causal.
In each layer, neurons are initialized with a certain time constant. We draw
these time constants from a Gaussian distribution with a given mean and
standard deviation of 20% of the mean. In this way, we introduce a slight
degree of heterogeneity in the SNN, although we observe minimal no beneficial
impact on the performance unlike [16]. This can be explained by the fact that
we do not feature recurrent connections, and thus do not exploit the recurrent
dynamics that is enriched by the heterogeneity of neurons. Note that temporal
hierarchy is formed by controlling the mean of the time-constant distribution
across the layers.
Multiple such layers are concatenated to form either a multi-layer spiking
perceptron (experiments in Figure 2) or a convolutional SNN (Figure 4). The
output layer, instead, is always made of Leaky-Integrators neurons, i.e. non-
spiking neurons. The time constant for such output neurons is
$200\text{\,}\mathrm{ms}$ for all experiments. The values of other relevant
hyperparameters can be found in Supplementary Table S1.

###  4.2 Loss function and regularization

#### Loss Function

Concerning the loss function, all experiments involve supervised
classification tasks, for which Cross-Entropy is chosen. The output logits are
computed as in [9], thus computing the softmax of the output neurons membrane
voltage at every time step, integrating these values over the whole duration
of the task, and applying the softmax once again on the obtained values. The
resulting loss function averages the cross-entropy over the task duration:

$\mathcal{L}_{\text{sum}}=-\frac{1}{T}\sum_{t}^{T}\log\left(\frac{\exp\left(\mathrm{out}[t,y]\right)}{\sum_{n}^{N}\exp\left(\mathrm{out}[t,n]\right)}\right)$

where $T$ is the task duration, $N$ number of output neurons,
$\mathrm{out}\in\mathcal{R}^{T,N}$ the output neurons membrane voltage, $y$
the desired label.
In the case of the MTS-XOR task, we opted for a “max-over time” loss function,
typical in many SNNs in the literature [24, 43]. This loss function looks for
the maximum value for the membrane voltage of the output neuron over a certain
period. In the case of the MTS-XOR, this period corresponds to the
presentation of the cues in Channel B. The "max-over time" loss function
equation is the following:

$\mathcal{L}_{\text{max}}=-\log\left(\frac{\exp(\mathrm{max}_{T}\left(\mathrm{out}[y])\right)}{\sum^{N}_{n}\exp(\mathrm{max}_{T}\left(\mathrm{out}[n])\right)}\right)$

where $\mathrm{max}_{T}$ indicates the selection of the maximum value of the
membrane voltage over the selection period $T$.

#### Regularization

Dropout with a probability of 10% is applied in the experiments in Figure 2
while the probability is increased at 40% in Figure 4. L2-regularization with
a magnitude of $10^{-4}$ is utilized for experiments with the convolutional
SNNs. When training the time constants of neurons (Fig.3), a regularization
term is applied to prevent unreasonable values. It is reminded that the time
constant has to be positive and that values that are too large might lead to
numerical instability. The resulting regularization term on the time constant
is a re-scaled L2-norm:

$\mathcal{L}_{\tau,L2}=\sum_{l}^{L}L2\left(\tau(l)-\mathrm{mean}[\tau(l)]\right)$

where $\tau(l)$ is the time constant at a given hidden layer, with L hidden
layers in total.

###  4.3 Data Augmentation

In some experiments on the convolutional SNN section (Table 1) we utilize data
augmentation to improve the generalization performance of the model. However,
in the spirit of this work, we looked for a simple yet effective strategy to
augment the SHD and SSC datasets. We don’t upsample the dataset but simply
transform its samples. In particular, the only transformation we apply is a
transposition of the input channels, equivalent to a frequency shift of the
speaker’s voice. For each data point, we draw a number with a mean of 10 and a
standard deviation of 5, representing the frequency shift of the data. We then
pad the input sequence with as many channels as the drawn frequency shift and
truncate the exceeding channels to recover the original dataset input size
(700 channels). The frequency shift can be positive or negative with 0.5
probability. Each sample in a batch of data has an independent frequency
shift.

##  5 Acknowledgment and Funding

We want to thank the EIS-Lab, especially Tristan Torchet for assisting with
the code implementation. The presented work has received funding from the
Swiss National Science Foundation Starting Grant Project UNITE
(TMSGI2-211461).

## References

  * Eshraghian et al. [2023] Jason K Eshraghian, Max Ward, Emre O Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D Lu.  Training spiking neural networks using lessons from deep learning.  _Proceedings of the IEEE_ , 2023.
  * Foffani et al. [2009] G Foffani, ML Morales-Botello, and J Aguilar.  Spike timing, spike count, and temporal information for the discrimination of tactile stimuli in the rat ventrobasal complex.  _Journal of Neuroscience_ , 29(18):5964–5973, 2009.
  * Gerstner and Kistler [2002] Wulfram Gerstner and Werner M Kistler.  _Spiking neuron models: Single neurons, populations, plasticity_.  Cambridge university press, 2002.
  * Bellec et al. [2018] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass.  Long short-term memory and learning-to-learn in networks of spiking neurons.  _Advances in neural information processing systems_ , 31, 2018.
  * Yin et al. [2023] Bojian Yin, Federico Corradi, and Sander M Bohté.  Accurate online training of dynamical spiking neural networks through forward propagation through time.  _Nature Machine Intelligence_ , 5(5):518–527, 2023.
  * D’Agostino et al. [2024] Simone D’Agostino, Filippo Moro, Tristan Torchet, Yiğit Demirağ, Laurent Grenouillet, Niccolò Castellani, Giacomo Indiveri, Elisa Vianello, and Melika Payvand.  Denram: neuromorphic dendritic architecture with rram for efficient temporal processing with delays.  _Nature communications_ , 15(1):3446, 2024.
  * Göltz et al. [2024] Julian Göltz, Jimmy Weber, Laura Kriener, Peter Lake, Melika Payvand, and Mihai A Petrovici.  Delgrad: Exact gradients in spiking networks for learning transmission delays and weights.  _arXiv preprint arXiv:2404.19165_ , 2024.
  * Hammouamri et al. [2023] Ilyass Hammouamri, Ismail Khalfaoui-Hassani, and Timothée Masquelier.  Learning delays in spiking neural networks using dilated convolutions with learnable spacings.  _arXiv preprint arXiv:2306.17670_ , 2023.
  * Bittar and Garner [2022] Alexandre Bittar and Philip N Garner.  A surrogate gradient spiking baseline for speech command recognition.  _Frontiers in Neuroscience_ , 16:865897, 2022.
  * Yin et al. [2020] Bojian Yin, Federico Corradi, and Sander M Bohté.  Effective and efficient computation with multiple-timescale spiking recurrent neural networks.  In _International Conference on Neuromorphic Systems 2020_ , pages 1–8, 2020.
  * Graves [2013] Alex Graves.  Generating sequences with recurrent neural networks.  _arXiv preprint arXiv:1308.0850_ , 2013.
  * Khajehabdollahi et al. [2023] Sina Khajehabdollahi, Roxana Zeraati, Emmanouil Giannakakis, Tim Jakob Schäfer, Georg Martius, and Anna Levina.  Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks.  _arXiv preprint arXiv:2309.12927_ , 2023.
  * Srivastava et al. [2015] Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber.  Training very deep networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Telgarsky [2016] Matus Telgarsky.  Benefits of depth in neural networks.  In _Conference on learning theory_ , pages 1517–1539. PMLR, 2016.
  * Weidel and Sheik [2021] Philipp Weidel and Sadique Sheik.  Wavesense: Efficient temporal convolutions with spiking neural networks for keyword spotting.  _arXiv preprint arXiv:2111.01456_ , 2021.
  * Perez-Nieves et al. [2021] Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman.  Neural heterogeneity promotes robust learning.  _Nature communications_ , 12(1):5791, 2021.
  * Zheng et al. [2024] Hanle Zheng, Zhong Zheng, Rui Hu, Bo Xiao, Yujie Wu, Fangwen Yu, Xue Liu, Guoqi Li, and Lei Deng.  Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics.  _Nature Communications_ , 15(1):277, 2024.
  * Harris et al. [2019] Julie A Harris, Stefan Mihalas, Karla E Hirokawa, Jennifer D Whitesell, Hannah Choi, Amy Bernard, Phillip Bohn, Shiella Caldejon, Linzy Casal, Andrew Cho, et al.  Hierarchical organization of cortical and thalamic connectivity.  _Nature_ , 575(7781):195–202, 2019.
  * Meunier et al. [2009] David Meunier, Renaud Lambiotte, Alex Fornito, Karen Ersche, and Edward T Bullmore.  Hierarchical modularity in human brain functional networks.  _Frontiers in neuroinformatics_ , 3:571, 2009.
  * Wacongne et al. [2011] Catherine Wacongne, Etienne Labyt, Virginie Van Wassenhove, Tristan Bekinschtein, Lionel Naccache, and Stanislas Dehaene.  Evidence for a hierarchy of predictions and prediction errors in human cortex.  _Proceedings of the National Academy of Sciences_ , 108(51):20754–20759, 2011.
  * Grafton and Hamilton [2007] Scott T Grafton and Antonia F de C Hamilton.  Evidence for a distributed hierarchy of action representation in the brain.  _Human movement science_ , 26(4):590–616, 2007.
  * Honey et al. [2012] Christopher J Honey, Thomas Thesen, Tobias H Donner, Lauren J Silbert, Chad E Carlson, Orrin Devinsky, Werner K Doyle, Nava Rubin, David J Heeger, and Uri Hasson.  Slow cortical dynamics and the accumulation of information over long timescales.  _Neuron_ , 76(2):423–434, 2012.
  * Murray et al. [2014] John D Murray, Alberto Bernacchia, David J Freedman, Ranulfo Romo, Jonathan D Wallis, Xinying Cai, Camillo Padoa-Schioppa, Tatiana Pasternak, Hyojung Seo, Daeyeol Lee, et al.  A hierarchy of intrinsic timescales across primate cortex.  _Nature neuroscience_ , 17(12):1661–1663, 2014.
  * Cramer et al. [2020] Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke.  The heidelberg spiking data sets for the systematic evaluation of spiking neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33(7):2744–2757, 2020.
  * Warden [2018] Pete Warden.  Speech commands: A dataset for limited-vocabulary speech recognition.  _arXiv preprint arXiv:1804.03209_ , 2018.
  * Stefanou et al. [2016] Stefanos S Stefanou, George Kastellakis, and Panayiota Poirazi.  Creating and constraining compartmental models of neurons using experimental data.  _Advanced Patch-Clamp Analysis for Neuroscientists_ , pages 325–343, 2016.
  * Masquelier [2018] Timothée Masquelier.  Stdp allows close-to-optimal spatiotemporal spike pattern detection by single coincidence detector neurons.  _Neuroscience_ , 389:133–140, 2018.
  * Moro et al. [2022] Filippo Moro, Emmanuel Hardy, Bruno Fain, Thomas Dalgaty, Paul Clémençon, Alessio De Prà, Eduardo Esmanhotto, Niccolò Castellani, François Blard, François Gardien, et al.  Neuromorphic object localization using resistive memories and ultrasonic transducers.  _Nature communications_ , 13(1):3506, 2022.
  * Göltz et al. [2021] Julian Göltz, Laura Kriener, Andreas Baumbach, Sebastian Billaudelle, Oliver Breitwieser, Benjamin Cramer, Dominik Dold, Akos Ferenc Kungl, Walter Senn, Johannes Schemmel, et al.  Fast and energy-efficient neuromorphic deep learning with first-spike times.  _Nature machine intelligence_ , 3(9):823–835, 2021.
  * Sun et al. [2023] Pengfei Sun, Ehsan Eqlimi, Yansong Chua, Paul Devos, and Dick Botteldooren.  Adaptive axonal delays in feedforward spiking neural networks for accurate spoken word recognition.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5, 2023.  doi:10.1109/ICASSP49357.2023.10094768.
  * Sun et al. [2022] Pengfei Sun, Longwei Zhu, and Dick Botteldooren.  Axonal delay as a short-term memory for feed forward deep spiking neural networks.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8932–8936, 2022.  doi:10.1109/ICASSP43922.2022.9747411.
  * Deckers et al. [2024] Lucas Deckers, Laurens Van Damme, Werner Van Leekwijck, Ing Jyh Tsang, and Steven Latré.  Co-learning synaptic delays, weights and adaptation in spiking neural networks.  _Frontiers in Neuroscience_ , 18:1360300, 2024.
  * Nowotny et al. [2022] Thomas Nowotny, James P Turner, and James C Knight.  Loss shaping enhances exact gradient learning with eventprop in spiking neural networks.  _arXiv preprint arXiv:2212.01232_ , 2022.
  * Schöne et al. [2024] Mark Schöne, Neeraj Mohan Sushma, Jingyue Zhuge, Christian Mayr, Anand Subramoney, and David Kappel.  Scalable event-by-event processing of neuromorphic sensory signals with deep state-space models.  _arXiv preprint arXiv:2404.18508_ , 2024.
  * Le et al. [2015] Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton.  A simple way to initialize recurrent networks of rectified linear units.  _arXiv preprint arXiv:1504.00941_ , 2015.
  * Pehle et al. [2022] Christian Pehle, Sebastian Billaudelle, Benjamin Cramer, Jakob Kaiser, Korbinian Schreiber, Yannik Stradmann, Johannes Weis, Aron Leibfried, Eric Müller, and Johannes Schemmel.  The brainscales-2 accelerated neuromorphic system with hybrid plasticity.  _Frontiers in Neuroscience_ , 16:795876, 2022.
  * Richter et al. [2024] Ole Richter, Chenxi Wu, Adrian M Whatley, German Köstinger, Carsten Nielsen, Ning Qiao, and Giacomo Indiveri.  Dynap-se2: a scalable multi-core dynamic neuromorphic asynchronous spiking neural network processor.  _Neuromorphic Computing and Engineering_ , 4(1):014003, 2024.
  * Orchard et al. [2021] Garrick Orchard, E Paxon Frady, Daniel Ben Dayan Rubin, Sophia Sanborn, Sumit Bam Shrestha, Friedrich T Sommer, and Mike Davies.  Efficient neuromorphic signal processing with loihi 2.  In _2021 IEEE Workshop on Signal Processing Systems (SiPS)_ , pages 254–259. IEEE, 2021.
  * Ramakrishnan et al. [2013] Shubha Ramakrishnan, Richard Wunderlich, Jennifer Hasler, and Suma George.  Neuron array with plastic synapses and programmable dendrites.  _IEEE transactions on biomedical circuits and systems_ , 7(5):631–642, 2013.
  * Wang and Liu [2010] Yingxue Wang and Shih-Chii Liu.  Multilayer processing of spatiotemporal spike patterns in a neuron with active dendrites.  _Neural computation_ , 22(8):2086–2112, 2010.
  * Izhikevich [2001] Eugene M Izhikevich.  Resonate-and-fire neurons.  _Neural networks_ , 14(6-7):883–894, 2001.
  * Bradbury et al. [2018] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs.  2018\.  URL http://github.com/google/jax.
  * Neftci et al. [2019] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke.  Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.  _IEEE Signal Processing Magazine_ , 36(6):51–63, 2019.
  * Yao et al. [2021] Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li.  Temporal-wise attention spiking neural networks for event streams classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10221–10230, 2021.
  * LeCun [1998] Yann LeCun.  The mnist database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.

## Supplementary Information

#### Table of Hyperparameters

We list in Table S1 the hyperparameters used in the experiments. The acronyms
and symbols refer to: learning rate (LR), feed-forward (FF), convolutional
(conv), Max-over-time loss function ($\mathcal{L}_{\mathrm{max}}$), sum-of-
softmax-output $\mathcal{L}_{\mathrm{sum}}$, output neuron time constant
$\tau_{\mathrm{out}}$, mean hidden layer time constant $\tau_{\mu}$.

|  | Batch Size | Epochs | Dropout | LR, decay [epochs start, decay] | Initialization | Loss |  $\tau_{\mu}$ [s] |  $\tau_{\mathrm{out}}$ [s]
---|---|---|---|---|---|---|---|---|---
MTS-XOR | FF | 512 | 60 | 0.1 | 0.01, linear [25, 0.5] | Xavier | $\mathcal{L}_{\mathrm{max}}$ | 0.3 | 0.2
SHD/SSC | FF | 256 | 60 | 0.1 | 0.01, linear [25, 0.5] | Xavier | $\mathcal{L}_{\mathrm{sum}}$ | 0.2 | 0.2
Conv | 256 | 100 | 0.4 | 0.01, linear [25, 0.5] | Xavier | $\mathcal{L}_{\mathrm{sum}}$ | 0.2 | 0.2

Table S1: Table of the hyperparameters.

### Feed Forward SNN with temporal hierarchy

Figure S1: a) Accuracy of SNNs with positive
($\Delta\tau=$+500\text{\,}\mathrm{ms}$$) and negative
($\Delta\tau=$-500\text{\,}\mathrm{ms}$$) hierarchy of time constant, as a
function of the background noise rate on the MTS-XOR task. Shaded areas show
the quartiles over 5 trials. b) Test accuracy on the SHD task for a network
with homogeneous time constants through the layers. c) Hierarchy shapes of the
$\tanh$ function that maximize classification accuracy. Error bars show the
quartiles over 5 trials in b) and c). d) The $\tanh$ function parameterized by
the 5 combinations of steepness and centering that yield the best results in
Figure 2f.

The intuition behind the role of temporal hierarchy is that the first hidden
layer acts as a filter, while deeper layers extrapolate higher-level features.
To prove this, we set up a simple experiment trying to perform the MTS-XOR
task with an increasing background noise rate. Figure S1a shows that SNNs with
positive hierarchy are more resilient to input background noise, while
negative hierarchy leads to a slightly larger dip in classification accuracy.
To assess the performance of temporal hierarchy in feed-forward SNNs, we first
optimize a conventional SNN by finding the optimal average time constant
$\tau_{\mu}$, homogeneous through the layers in this case. Results of this
characterization on the SHD dataset are shown in Supplementary Figure S1b,
where classification accuracy is maximized in the range
$\tau_{\mu}\in[$100\text{\,}\mathrm{ms}$,$200\text{\,}\mathrm{ms}$]$. For the
rest of the experiments, we thus assume
$\tau_{\mu}=$200\text{\,}\mathrm{ms}$$.
While in Figure 2e we analyzed the effect of the shape of time constant
hierarchy, in Supplementary Figure S1c we look at the effect of the amplitude
$\color[rgb]{0,0.88,0}{\Delta\tau}$. As previously observed in Figure 2d,
there is a correlation between classification accuracy and temporal hierarchy
amplitude, which can provide a boost of performance of up to 4.1%.
Based on the analysis of the shape of hierarchy (Figure 2e), we plot the
combinations of steepness and centering that optimize the performance in
Supplementary Figure S1c.

### State-of-the-art SNNs in temporal tasks

To better compare the performance of SNNs endowed with temporal hierarchy, we
compile a table of results (Table S2) on the Spiking Heidelberg Digit dataset
[24]. This table is based on the resources in the SHD original website
https://zenkelab.org/resources/spiking-heidelberg-datasets-shd/. The scope of
the table is to highlight the role of the different computational principles
and inductive biases on the performance of SNNs. Note that this table only
includes spike-based algorithms. Other neural network architectures (LSTMs
[24], CNNs, and event-based State Space Models [34]) have been tested on this
task.
Hierarchy improves the performance of SNNs (hSNN), with the best result for
hSNN less than 1% away from the state-of-the-art performance [8] despite
featuring simple LIF-based convolutional SNNs. Note that in this work we
wanted to isolate the effect of temporal hierarchy, thus the choice of basic
SNN architectures, avoiding other features like recurrence [24], adaptation
[9], temporal attention, and complex neuron models.
We believe temporal hierarchy can easily be adopted by other researchers and
improve the computational power of their models.

| Network Features |  data augment. |  Test as Valid. | Network Architecture | Accuracy
---|---|---|---|---|---
| Rec | Adapt | Att |  | delays/
---
conv.
hierarchy |  | # hid.
---
layers
| hidden
---
size
# params | %
| Perez-Nieves et al. [16]
---
✓ |  |  |  |  |  |  | 1 | 256 | 100k | 82.7 ± 0.8
| D’Agostino et al. [6]
---
|  |  | ✓ |  |  |  | / | / | 224k | 90.7
| Yao et al. [44]
---
|  | ✓ |  |  |  |  | 2 | 128 | / | 91.1
| Sun et al. [30]
---
|  |  | ✓ |  |  |  |  |  | 110k | 92.5
| Nowotny et al. [33]
---
✓ |  |  |  |  | ✓ |  | 3 | 256 | 250k | 93.5 ± 0.7
| Bittar and Garner [9]
---
✓ | ✓ |  |  |  |  | ✓ | 2 | 1024 | 1.8M | 94.6
| Hammouamri et al. [8]
---
|  |  | ✓ |  |  | ✓ | 2 | 256 | 200k | 95.1 ± 0.3
hSNN [ours] |  |  |  |  | ✓ |  |  | 4 | 32 | 25k | 84.9 ± 1.6
|  |  |  | ✓ |  |  | 4 | 128 | 150k | 88.1 ± 0.6
hCSNN [ours] |  |  |  | ✓ | ✓ |  |  | 4 | 128 | 345k | 92.2 ± 1.1
|  |  | ✓ | ✓ | ✓ |  | 94.1 ± 0.7
|  |  | ✓ | ✓ | ✓ | ✓ | 94.7 ± 0.7

Table S2: Summary of the performance of SNNs with and without temporal
hierarchy and comparison with the state-of-the-art in the SHD dataset.
Abbreviations: Rec for recurrent, Adapt for neuronal adaptation, Att for
temporal attention, conv. for temporal convolutions, “Test as Valid.” refers
to utilizing the test set as the validation set for hyperparameter
optimization, which is considered bad practice in machine learning but
employed here once nevertheless to compare with other papers [9, 34].

### Optimization of the time constants discovers temporal hierarchy

We extend here the results of the optimization of the time constant, in Figure
3. This time, we train a 5-hidden-layer SNN initialized with a homogeneous
time constant of $\tau_{\mu}=$200\text{\,}\mathrm{m}\mathrm{s}$$ on the
Spiking Heidelberg Digit dataset. The resulting probability density functions
averaged over 5 trials, are shown in Supplementary Figure S2. Optimization
produced nicely separated PDFs for the time constants of the neurons in each
hidden layer.

Figure S2: Optimization of the time constant training SNNs on the SHD task.
The probability density function (PDF) of the time constants of neurons is
shown in each of the 5 hidden layers (green). The mean value in each layer
(purple) reveals the formation of hierarchy in the time constant. Results were
collected by averaging from 5 trials.

### SNNs with temporal causal convolutions

As a preliminary analysis of temporal causal convolutional SNNs, the optimal
kernel size and dilation are found via hyper-parameter optimization. In this
experiment, data augmentation is disabled. In Figure S3 (left), we fix the
dilation to 5 and vary the kernel size between 3 and 9. A kernel size of 5
optimizes the performance of the convolutional SNN. Similarly, Figure S3
(right) shows the classification accuracy as a function of the dilation, when
the kernel size is fixed to 5. Despite a dilation of 3 optimizes performance,
we fix the dilation to 5 for the rest of the experiments in Section 2.3. This
allows more control of the temporal hierarchy produced with dilations through
the hidden layers.

Figure S3: Hyperparameter tuning of the convolutional SNNs. We experiment on
the effect of kernel size on the left, of the dilation on the right. Error
bars show the quartiles over 5 trials.

### Temporal hierarchy with static data

While temporal hierarchy is demonstrated as a positive Inductive Bias that
improves the performance in SNNs with tasks that feature information in the
temporal domain, we investigate here what happens when the data is static in
time, i.e. there is no relevant temporal dynamics at the input. To analyze
this, temporal hierarchy is tested with the ever-green MNIST dataset [45], a
simple image classification dataset. As the data is provided as static images,
we add a fictitious temporal dimension via Latency Coding (Figure S4a). This
means the images are first flattened to 1 dimension (of size 784). Then the
brightness of each pixel is converted to the latency of an event (or spike)
that occurs at the input. Each image, originally in gray-scale with $28\times
28$ size, is transformed to size $784\times T$, where $T$ is the number of
time steps for the Latency Coding. For our experiments, we set $T=50$ steps,
with each step $dt=$20\text{\,}\mathrm{ms}$$, thus showing each image for 1
second. Note that summing along the fictitious temporal dimension would allow
us to recover the information from the original image, provided that the
precision of the input pixel brightness (8b) matches the precision of the
temporal dimension ($T=2^{8}=256$). In this case, then, the input images are
slightly compressed.
We first analyze the effect of setting the time constant homogeneously through
the 2 hidden layers of a Spiking Neural Network. In Figure S4b we observe
that, according to the expectation, the task can be solved by “integrating”
the information at the input. Thus, the larger the time constant the better.
Setting $\tau_{\mu}=$500\text{\,}\mathrm{ms}$$, we then impose a hierarchy of
time constant with amplitude $\Delta\tau$ (Fig. S4c). Once again, changing the
time constants does not impact the classification accuracy. This confirms once
again that it is sufficient to integrate the information over time to solve
this task. If the time constant of the neuron is larger than some threshold
value (around $200\text{\,}\mathrm{ms}$ in this case), imposing a structure to
the time constant does not affect the computation of the SNN. Temporal
hierarchy is then a positive inductive bias for temporal tasks only.

Figure S4: Role of temporal hierarchy as time constant hierarchy on static
datasets. a) MNIST is converted with Latency Coding to introduce the temporal
dimension to the data. b) Classification accuracy as a function of the mean
homogeneous time constant $\tau_{\mu}$, same for all layers. c) Classification
accuracy when time constant hierarchy is introduced, as a function of the
hierarchy amplitude $\Delta\tau$. Error bars show the quartiles over 5 trials.

It is well-known that MNIST can be transformed into a sequential task, where
the order of the sequence encodes information [35]. Presenting the MNIST
images as a 1-dimensional sequence (Figure S5) offers a challenge for a
classifying model, that has to encode the sequence into working memory to be
able to classify it correctly. In this setting, temporal hierarchy should play
a role, as it improves the temporal processing performance of SNNs. We test
the hypothesis with an SNN composed of 2 layers with 64 LIF neurons each. Time
is fictitious in this task, and we assign the sequence length (784 steps) to a
duration of $1\text{\,}\mathrm{s}$. The mean time constant $\tau_{\mu}$ of
neurons is set 50 time-steps, corresponding to about $60\text{\,}\mathrm{ms}$.
Based on this network configuration, temporal hierarchy is introduced as the
time constant difference $\Delta\tau$ in the initialization. The results of
varying $\Delta\tau$ between $-100\text{\,}\mathrm{ms}$ and
$+100\text{\,}\mathrm{ms}$ are shown in Figure S5b: we note the positive
correlation between $\Delta\tau$ and classification accuracy. However,
accuracy improved also $\Delta\tau=$-100\text{\,}\mathrm{ms}$$. A possible
explanation is that some longer time scales are still beneficial in solving
this task, as evidenced in [4]. Similarly, we repeated the same experiment but
with enabled recurrent connections (Figure S5c). Classification accuracy had a
positive impact, while the overall behavior related to the time-constant
hierarchy remained.

Figure S5: Role of temporal hierarchy as time constant hierarchy on the
sequential MNIST task. a) MNIST is converted to a 1-dimensional sequence, with
time stamps of 1 ms each. b) Classification accuracy when time constant
hierarchy is introduced, as a function of the hierarchy amplitude
$\Delta\tau$, in a feed-forward SNN. c) Same as b), but with enabled recurrent
connections. Error bars show the quartiles over 5 trials.

  *[SNNs]: Spiking Neural Network
  *[LIF]: Leaky Integrate and Fire
  *[SHD]: Spiking Heidelberg Digit
  *[SNN]: Spiking Neural Network