# Robot Subset Selection for Swarm Lifetime Maximization in Computation
Offloading with Correlated Data Sources

Siqi Zhang, Na Yi, and Yi Ma†
5GIC and 6GIC, Institute for Communication Systems, University of Surrey,
Guildford, UK, GU2 7XH
Emails: (s.zhang, n.yi<EMAIL_ADDRESS>
###### Abstract

Consider robot swarm wireless networks where mobile robots offload their
computing tasks to a computing server located at the mobile edge. Our aim is
to maximize the swarm lifetime through efficient exploitation of the
correlation between distributed data sources. The optimization problem is
handled by selecting appropriate robot subsets to send their sensed data to
the server. In this work, the data correlation between distributed robot
subsets is modelled as an undirected graph. A least-degree iterative
partitioning (LDIP) algorithm is proposed to partition the graph into a set of
subgraphs. Each subgraph has at least one vertex (i.e., subset), termed
representative vertex (R-Vertex), which shares edges with and only with all
other vertices within the subgraph; only R-Vertices are selected for data
transmissions. When the number of subgraphs is maximized, the proposed subset
selection approach is shown to be optimum in the AWGN channel. For independent
fading channels, the max-min principle can be incorporated into the proposed
approach to achieve the best performance.

##  I Introduction

By robot swarm, we refer to the definition provided by the US Army Academy [1,
Sec. II]:

######  Definition 1 (Robot Swarm)

A robot swarm is a group of (three or more) robots that perform tasks
cooperatively while receiving limited or no control from human operators. The
term ‘cooperatively’ is defined in the way that involves mutual assistance in
working towards a common goal. It does not necessarily imply or require
communication or explicit coordination between entities.

This technology has found rich applications including disaster recovery,
defense, reconnaissance, inspection, mapping, farming, food management, and
space systems.

In light of Definition 1, we consider a robot swarm where a set of mobile
robots cooperatively perform a single task. Every robot can conduct
environment sensing and if required send their sensed data to a computing
server through a connected access point (AP). The server conducts processing
on the received data and sends computing outcomes to the robots. This work
falls into the scope of mobile edge computing (MEC) in the wireless
communication domain. The motivation of offloading computing tasks to the edge
server lies in (see [2]): 1) data processing (computing) tasks are often
energy hungry and thus not suitable for battery-powered mobile robots; 2) edge
servers are often equipped with powerful computing units (such as GPUs, APUs
or NPUs), which can provide parallel and low-latency computing services.

Our prior arts analysis shows that most of current MEC computation-task
offloading works are focused on resource allocation problems in single-user
equivalent computing scenarios (e.g. [3, 4, 5, 6]), i.e., despite multi-user
communications in reliability, computing tasks (or data) from different
devices are processed individually at the edge server. Recently, multi-user
edge computing has attracted increasing interests, where data from different
devices must be jointly processed to yield the computing outcome. In this
case, users are branch and bound in the procedure of computation-task
offloading [7].

This work falls into the scope of multi-user edge computing, with the aim of
maximizing the lifetime of robot swarm exploiting the data source correlation
between mobile robots. Major contributions of this work include: 1) the
development of a novel robot-subset model using an undirected graph,
describing the data source correlation between spatially distributed robots;
2) the establishment of a novel cost function, formulating the optimization
problem for the swarm-lifetime maximization; 3) the development of a novel
graph partitioning and subset selection approach to tackle the swarm-lifetime
maximization problem. It is shown that, when the number of subgraphs is
maximized, the proposed subset selection approach is optimum in the AWGN
channel. As far as independent flat-fading channels are concerned, the max-min
principle can be incorporated into the proposed approach to maximize the swarm
lifetime. The above findings are elaborated through well-designed computer
simulations.

##  II System Model and Problem Formulation

In this section, we introduce the basic concept of robot subset, data
correlation, communication model, as well as the formulation of optimization
problem.

###  II-A Robot, Subset, and Data Correlation Model

Consider a robot swarm where a set of mobile robots ($N$) collaboratively
perform a single task. Mathematically, a robot is defined as a tuple,
$s\triangleq(x,\mathcal{E},\varepsilon)$, where $x$ stands for the sensed
data, $\mathcal{E}$ for the remaining energy of the robot, and $\varepsilon$
for the energy consumption per job 111For the sake of concise presentation,
$\varepsilon$ is assumed to be a constant.. Then, the robot swarm is described
as a set: $\mathcal{S}\triangleq\\{s_{n}|n=1,...,N\\}$. Provided that all
robots send their data to the MEC server, the multi-user edge computing
process can be described as

$y=f(x_{n}|n=1,...,N),~{}y\in\mathcal{A}\triangleq\\{a_{l}|l=1,...,L\\},$ (1)

where $x_{n}$, ${}_{n=1,...,N},$ denotes the data from the $n^{th}$ robot, and
$y$ denotes the computing outcome. Following Definition 1, we assume there is
no human intervention in the computing and decision making. Then, the outcome
$(y)$ must belong to a finite-alphabet set defined by $\mathcal{A}$, which
specifies all possible actions (i.e., $a_{l},_{l=1,\cdots,L}$) of the robot
swarm; $L$ is the total number of actions. Since $(\mathcal{E},\varepsilon)$
has no impact on $y$, (1) can have the following notation-simplified form

$y=f(\mathcal{S}),~{}y\in\mathcal{A}.$ (2)

The above description is for the generic case. However, in many practical
applications, the MEC does not need the whole set ($\mathcal{S}$) to form the
computing outcome. This is because robots’ data are correlated and carry
considerable redundancy. This insight motivates us to propose the concept of
robot subset in the multi-user edge computing.

######  Definition 2 (Robot Subset)

Consider a subset of robots within the swarm, denoted by
$\Omega_{m}\subset\mathcal{S}$ with $|\Omega_{m}|=K_{m}$, m=1,…,M, where
$|\cdot|$ stands for the cardinality of the set, and $M$ for the number of
subsets. The MEC outcome corresponding to the subset $\Omega_{m}$ is described
by

$y_{m}=f(\Omega_{m}),~{}y_{m}\in\mathcal{A}.$ (3)

It is assumed that there exists a $K(<N)$, for $K_{m}\geq K$, the result
$(y_{m}=y)$ always holds 222This assumption turns out to be true for robot (or
sensor) networks where robots (or sensors) are closely located in a
geographical area (see [8]).. Using the terminology of data science [9], the
robot subset ($\Omega_{m}$) is said to have the same value of information as
the whole set $(\mathcal{S})$.

Definition 2 implies that the whole set ($\mathcal{S}$) can be divided into
$M$ robot subsets, with each having the same value of information. Then, for
each computing task, only one of subsets is required to send their sensed data
to the MEC. This will result in significant saving in communication energy and
longer swarm lifetime. The issue of how to divide $\mathcal{S}$ into subsets
will be addressed in Section IV-A.

###  II-B Communication Model

For every single task, the MEC chooses one of robot subsets (denoted by
$\Omega_{m}$), requesting them send their data to the MEC through
$|\Omega_{m}|$ orthogonal sub-channels. Without loss of generality, we assume
$|\Omega_{m}|=K,{\forall m}$. For a robot $s_{k}\in\Omega_{m}$, the channel
capacity between this robot and the AP is denoted as $C(\alpha_{k},p_{k})$,
where $\alpha_{k}$ is the channel quality, and $p_{k}$ is the transmit power
which is capped by a physical constraint $(p_{k}\leq P_{k})$. Then, the robot
subset ($\Omega_{m}$) must fulfill the following criteria:

Criterion 1: $\forall s_{k}\in\Omega_{m}$, their transmission rate (denoted by
$R_{k}$) must fulfill

$R_{k}<C(\alpha_{k},p_{k})\leq C(\alpha_{k},P_{k});$ (4)

or otherwise the robot ($s_{k}$) or the subset ($\Omega_{m}$) must not be
chosen due to the capacity outage.

Criterion 2: The data transmission must be completed within a time duration
($T_{\textsc{c}}$) for the sake of latency.

Criterion 3: All robots must have enough remaining energy to execute the
allocated job.

After the computing process (3), the outcome ($y$) is broadcasted to all
robots. It is assumed that the broadcast is reliable and efficient. This is
the commonly used assumption in the literature (e.g., [3, 10, 11]), which
allows us focusing on the key problem of interest.

###  II-C Optimization Problem

This work aims to maximize the swarm lifetime through optimum selection of
robot subsets for the data transmission. The swarm lifetime is defined as:

######  Definition 3 (Swarm Lifetime)

A task assigned to the robot swarm requires collaborative work amongst all the
robots. Should any robot stop working due to the short of device energy
results in task failure of the whole robot swarm. Therefore, the swarm
lifetime counts as the maximum number of tasks a robot swarm can complete all
the way from the beginning to the end.

Consider the life cycle of the $i^{th}$ task conducted by the robot swarm.
Robots have the following energy consumption model:

$\mathcal{E}_{k}(i)=\mathcal{E}_{k}(i-1)-p_{k}(i)T_{\textsc{c}}-\epsilon-\varepsilon,~{}\forall
s_{k}\in\Omega(i),$ (5a)
$\mathcal{E}_{\ell}(i)=\mathcal{E}_{\ell}(i-1)-\varepsilon,~{}\forall
s_{\ell}\in\overline{\Omega(i)},$ (5b)

where $\epsilon$ is the energy consumption for communication-related
computation such as the source and channel coding, and the subscript
$(\cdot)_{\ell}$ is the index for unselected robots. Note that we shall have
the condition:

$\mathcal{E}_{k}(i)\geq 0,\mathcal{E}_{\ell}(i)\geq 0,\forall k,\ell;$ (6)

or otherwise, the $i^{th}$ task won’t happen, and the $(i-1)^{th}$ task would
be the last task assigned to the robot swarm. Therefore, whether or not we
will have the $i^{th}$ task, this depends on the strategy of robot selection
in the previous tasks.

Define $\Phi(i-1)\triangleq(\Omega(1),\Omega(2),...,\Omega(i-1))$ with
$\Phi(0)=\emptyset$. We further define $\Gamma(\Phi(i-1))$ the strategy of
robot selection which yields

$\Gamma(\Phi(i-1))=\left\\{\begin{array}[]{cc}i,&(\ref{eqn06})~{}\mathrm{is~{}true}\\\
0,&(\ref{eqn06})~{}\mathrm{is~{}false}\end{array}\right.$ (7)

According to Definition 3, the swarm-lifetime maximization problem has the
following objective function

$\Phi^{\star}(i-1)=\underset{\Phi(i-1)}{\arg\max}~{}\Gamma(\Phi(i-1)).$ (8)

This is the optimization problem we strive to address in Section III.

In the scope of wireless sensor networks (WSNs), the problem of system
lifetime maximization has already been extensively studied; see the survey
paper [12] and the references therein. However, the lifetime of WSN is very
different from that of robot swarm in manifold: 1) WSNs have no task to
execute, i.e., the term $\varepsilon$ won’t appear in the energy cost function
(5); 2) the life of WSN continues when one of sensors dies, i.e., the
condition (6) does not necessarily hold; 3) sensors often have their data
highly or fully correlated, and thus the lifetime maximization problem can be
handled through a sensor-level selection protocol; there is no multi-user edge
computing problem involved. All of these differences render our robot swarm
lifetime maximization a novel problem of optimization.

##  III The Method for Robot Swarm Lifetime Maximization

###  III-A Analysis of The Optimization Problem

The implication of (8) is that the lifetime maximization is equivalent to the
convergence problem of an iterative algorithm. The only difference is that we
want the convergence to be as slow as possible. Specifically, (5b) can be
written into

$\mathcal{E}_{\ell}(i)=\mathcal{E}_{\ell}(0)-i\varepsilon,~{}\forall
s_{\ell}\in\overline{\Omega}.$ (9)

It means that $\mathcal{E}_{\ell}(i)$ features linear convergence with the
rate of $\varepsilon$. A bit complicated case is (5a), where $E_{k}(i)$ does
not converge linearly. Nevertheless, for every single robot, we can present
their energy consumption in an universal model

$\mathcal{E}_{n}(i)=\mathcal{E}_{n}(0)-\zeta_{n}(i)-i\varepsilon,$ (10)

and $\zeta_{n}(i)$ is defined by

$\zeta_{n}(i)\triangleq\sum_{i^{\prime}=0}^{i}\sum_{j\in\Psi_{n}}(p_{k}(i^{\prime})T_{\textsc{c}}+\epsilon)\delta(i^{\prime}-j),$
(11)

where $\Psi_{n}$ is the set collecting all iteration indexes when the $n^{th}$
robot is chosen to send their data, and $\delta(\cdot)$ is the Dirac Delta
function [13].

Define $e_{n}(i)\triangleq\mathcal{E}_{n}(0)-\zeta_{n}(i)$. A robot with
smaller $e_{n}(i)$ has a shorter lifetime. According to Definition 3, the
swarm lifetime is determined by the robot with the shortest lifetime.
Therefore, the problem of swarm lifetime maximization is equivalent to
maximizing the minimum of $e_{n}(i),\forall n$, i.e.,

$\Phi^{\star}(i-1)=\underset{\Phi(i-1)}{\arg\max}\Big{(}\min(e_{1}(i),e_{2}(i),...,e_{N}(i))\Big{)}.$
(12)

The max-min optimization problem in (12) is well-known as NP-hard (see [14]).
However, it is possible to find good sub-optimum solutions based on some
relaxed conditions. We will tackle this problem step by step all the way from
the AWGN channel to the fading channel.

###  III-B Optimization in The AWGN Channel

In the context of AWGN channel, the channel quality measure, $\alpha_{n}$, is
not robot dependent. Instead, it can be simply set as: $\alpha_{n}=1,\forall
n$. Then, the channel capacity is not a criterion for selecting a robot
subset. The key parameters are mainly the remaining energy $\mathcal{E}_{n}$
and the subsets $\Omega_{m},\forall n,m$. Next, we will discuss about two
cases when robots have or do not have identical initial state of the remaining
energy.

####  III-B1 The case with identical state of the remaining energy

###### Lemma 1

Suppose: c1) $\mathcal{E}_{n}(0),\forall n,$ are identical; c2)
$\Omega_{c}\triangleq\Omega_{1}\cap\Omega_{2}\cdots\cap\Omega_{M}\neq\emptyset$.
The swarm lifetime ($i$) is given by

$i=\left\lfloor\frac{\mathcal{E}(0)}{pT_{\textsc{c}}+\epsilon+\varepsilon}\right\rfloor,$
(13)

where $\mathcal{E}$ and $p$ have their index omitted since they are identical
for all robots, and $\lfloor\cdot\rfloor$ stands for the integer floor.

###### Proof:

Given the condition c1), robot subsets are the only influencing factors for
the lifetime maximization. Given the condition c2), no matter which subset is
chosen, the common part of all subsets ($\Omega_{c}$) is always chosen.
Therefore, robots within $\Omega_{c}$ have the shortest lifetime, and their
energy consumption model (10) becomes

$\mathcal{E}(i)=\mathcal{E}(0)-i(pT_{\textsc{c}}+\epsilon+\varepsilon).$ (14)

Given $\mathcal{E}(i)\geq 0$, solving this inequality leads to (13). ∎

###### Lemma 2

Given the conditions c1) and c3)
$\Omega_{m_{1}}\cap\Omega_{m_{2}}=\emptyset,\forall m_{1}\neq m_{2}$, the
swarm lifetime ($i$) is upper-bounded by

$i\leq\frac{\mathcal{E}(0)-\left\lceil(i)/(M)\right\rceil(pT_{\textsc{c}}+\epsilon)}{\varepsilon},$
(15)

where $\lceil\cdot\rceil$ stands for the integer ceiling.

###### Proof:

With the conditions c1) and c3), the best strategy for subset selection is
round robin because all subsets are equal. In this case, the energy
consumption model generally stays the same as (10), but the term
$\zeta_{n}(i)$ becomes a robot-index independent term

$\zeta(i)=\left\lceil\frac{i}{M}\right\rceil(pT_{\textsc{c}}+\epsilon).$ (16)

This expression describes the subsets which have been chosen for the most of
times. In other words, robots within those subsets have the shortest lifetime.
Due to $\mathcal{E}(i)\geq 0$, we immediately have the inequality (15). The
upper bound (15) does not offer a closed-form solution for $i$. Nevertheless,
it is very easy to determine $i$ through line searching. ∎

Lemma 1 & 2 exhibits the maximum swarm lifetime only for two special cases of
robot subsets. Nevertheless, they lay the foundation for us to address the
lifetime maximization in a generic case, for which we model the subset
selection as a graph partitioning problem.

######  Definition 4 (Graphical Model of Subsets)

Define an undirected graph $G(V,E)$, where $V$ denotes the set of vertices and
$E$ the set of edges. There are $M$ vertices in the graph (i.e., $|V|=M$)
which are corresponding to $M$ robot subsets. The edge between two vertices
reflects the presence of common part between two corresponding subsets (i.e.,
the edge between the vertices $m_{1}$ and $m_{2}$ exists when
$\Omega_{m_{1}}\cap\Omega_{m_{2}}\neq\emptyset$). A degree matrix $\mathbf{D}$
is formed to record the number of edges linked to vertices (see the definition
of $\mathbf{D}$ in the graph theory [15]).

######  Definition 5 (Graph Partitioning Problem)

Our graph partitioning problem is stated by: partitioning an undirected graph
into a maximal number of subgraphs ($\overline{M}$) fulfilling the criterion:
c4) each subgraph has at least one vertex that shares edges with and only with
all other vertices within the subgraph. We call this vertex representative
vertex (R-Vertex).

According to Definition 4, the condition c2) can now be described by a
complete graph $G(M,\frac{(M)(M-1)}{2})$, and the condition c3) can be
described by a null graph $G(M,0)$. For the complete graph, we cannot further
partition the graph, and thus have $\overline{M}=1$. For the null graph, it is
naturally partitioned into $\overline{M}=M$ subgraphs. For both cases, any
vertex is a R-Vertex. When conducting subset selection at the subgraph level,
it is trivial to find that the swarm lifetime is determined by (14) and (15),
respectively. Applying the graph partitioning concept onto the generic case,
the following result can be obtained.

###### Theorem 1

For the generic case when the conditions c1) and c4) hold, the swarm lifetime
is upper-bounded by

$i\leq\frac{\mathcal{E}(0)-\left\lceil(i)/(\overline{M})\right\rceil(pT_{\textsc{c}}+\epsilon)}{\varepsilon}.$
(17)

###### Proof:

Suppose that a graph can be maximally partitioned into $\overline{M}$
subgraphs following the criterion specified in c4). Subset selection is
applied on the subgraph level in a round robin manner. For each subgraph, only
(one of) the R-Vertex (R-Vertices) is chosen for the data transmission. Since
R-Vertices from different subgraphs are not directly connected to each other,
this is the equivalent case to that in Lemma 2. Following the discussion in
the proof of Lemma 2, the swarm lifetime is determined by (17).

The reason of choosing R-Vertices is to ensure that there is no overlap
between selected subsets; or otherwise, some robots would be chosen multiple
times in one round, and such will shorten the swarm lifetime; as have already
been discussed in Lemma 2. Given $\overline{M}$ being the maximum number of
subgraphs, the right-hand side of (17) is maximized. Theorem 1 is therefore
proved. ∎

Theorem 1 shows that the key for lifetime maximization is to find
$\overline{M}$ through an optimum graph-partitioning algorithm. However, graph
partitioning is usually a NP-hard problem, and we can hardly claim an optimum
solution [16]. In this paper, we propose a least-degree iterative partitioning
(LDIP) algorithm that can offer a good sub-optimum solution for the graph
partitioning.

Fig. 1 illustrates the concept of the LDIP algorithm. Basically, the algorithm
consists of three steps:

Step 1 (Least degree finding): Sort the diagonal of the degree matrix
$\mathbf{D}$ in an ascent order; the first diagonal entry of $\mathbf{D}$
(denoted by $D_{(0,0)}$) has the least degree.

Step 2 (Subgraph forming): The vertex (subset) corresponding to $D_{(0,0)}$ is
chosen to be the R-Vertex of the subgraph under construction. All vertices
that are directly connected to the R-Vertex are included into this subgraph.

Step 3 (Degree matrix updating): Update the degree matrix $\mathbf{D}$
(dimensional reduction) by eliminating all vertices that have been chosen at
Step 2. Repeat Step 1 until $\mathbf{D}$ reduces to a scalar equaling to 0,
i.e., there is no vertex left.

Figure 1: The concept of the LDIP algorithm for graph partitioning. Red
ellipses are subgraphs; blue-faced circles are R-Vertices of their subgraphs.

The sub-optimality of LDIP lies in the fact that there may exist multiple
vertices sharing the least degree at each iteration. An optimum solution
requires to visit all possible cases for each iteration, resulting in
exponential searching complexity. Instead, LDIP randomly chooses a vertex at
each iteration, offering linear searching complexity at the price of
optimality.

####  III-B2 The case with non-identical state of the remaining energy

Consider the condition: c5) $\mathcal{E}_{n}(0),\forall n$, can be different,
the lifetime maximization problem becomes much more complicated because it
involves multiple parameters (i.e., $\mathcal{E}_{n}(0)$ and $\Omega_{m}$) in
the optimization. The round-robin algorithm is no longer optimum since subsets
should not be equally treated. Again, we start our study from the two special
cases described in Lemma 1 & 2.

###### Theorem 2

Provided the conditions c2) and c5), the swarm lifetime ($i$) is upper-bounded
by

$i\leq\left\lfloor\frac{\min(\mathcal{E}_{\Omega_{c}}(0))}{pT_{\textsc{c}}+\epsilon+\varepsilon}\right\rfloor,$
(18)

where $\min(\mathcal{E}_{\Omega_{c}}(0))$ is corresponding to the robot in the
set $\Omega_{c}$ who has the minimum $\mathcal{E}(0)$. A necessary condition
for this upper bound to be achievable is

$\left\lfloor\frac{\min(\mathcal{E}_{\Omega_{c}}(0))}{pT_{\textsc{c}}+\epsilon+\varepsilon}\right\rfloor\leq\sum_{m=1}^{M}\left\lfloor\frac{\min(\mathcal{E}_{\Omega_{m}\setminus\Omega_{c}}(0))-i\varepsilon}{pT_{\textsc{c}}+\epsilon}\right\rfloor.$
(19)

###### Proof:

We start by proving the upper bound (18). Following the proof of Lemma 1, no
matter which robot subset is chosen, robots within the set $\Omega_{c}$ will
always be selected, and they have the energy consumption model like (14),
i.e.,

$\mathcal{E}_{\ell}(i)=\mathcal{E}_{\ell}(0)-i(pT_{\textsc{c}}+\epsilon+\varepsilon),\ell\in\Omega_{c}.$
(20)

Then, the lifetime of $\Omega_{c}$ is limited by the robot with the minimum
$\mathcal{E}(0)$, which has the form (18). According to Definition 3, the
swarm lifetime cannot go beyond this upper limit.

We further consider robots in the set $\Omega_{m}\setminus\Omega_{c},\forall
m$. Their energy consumption model is given by

$\mathcal{E}_{k}(i)=\mathcal{E}_{k}(0)-w_{m}(pT_{\textsc{c}}+\epsilon)-i\varepsilon,~{}k\in\Omega_{m}\setminus\Omega_{c},$
(21)

where $w_{m}$ denotes how many times the robot subset is chosen. Then, the
lifetime of the robot set $\Omega_{m}\setminus\Omega_{c}$ is given by

$w_{m}=\left\lfloor\frac{\min(\mathcal{E}_{\Omega_{m}\setminus\Omega_{c}}(0))-i\varepsilon}{pT_{\textsc{c}}+\epsilon}\right\rfloor.$
(22)

Consider an optimistic case when 333When (23) does not hold, the common part
of two sets will be called whenever one of them are called. This can reduce
the lifetime of the union of the two sets.

$(\Omega_{m_{1}}\setminus\Omega_{c})\cap(\Omega_{m_{2}}\setminus\Omega_{c})=\emptyset,\forall
m_{1}\neq m_{2},$ (23)

the lifetime for the union set
$(\Omega_{1}\setminus\Omega_{c})\cup(\Omega_{2}\setminus\Omega_{c})\cup\cdots(\Omega_{M}\setminus\Omega_{c})$
is $\sum_{m=1}^{M}w_{m}$. This lifetime must be no less than the upper bound
(18); or otherwise, the upper bound is not achievable. Theorem 2 is therefore
proved. ∎

Note that the proof of Theorem 2 has already incorporated the condition c3),
which is equivalent to (23) with $\Omega_{c}=\emptyset$. Hence, for the
condition c3), we have

$w_{m}=\left\lfloor\frac{\min(\mathcal{E}_{\Omega_{m}}(0))-i\varepsilon}{pT_{\textsc{c}}+\epsilon}\right\rfloor$
(24)

and reaches the following conclusion:

###### Corollary 2.1

Given the conditions c3) and c5), the swarm lifetime ($i$) is upper-bounded by

$i\leq\sum_{m=1}^{M}\left\lfloor\frac{\min(\mathcal{E}_{\Omega_{m}}(0))-i\varepsilon}{pT_{\textsc{c}}+\epsilon}\right\rfloor.$
(25)

The technical insight from Theorem 2 and Corollary 2.1 is that robot subsets
must be weighted in the procedure of subset selection. Their weights can be
quantified by $w_{m}$ (see (22) and (24)), which are related to the terms
$\min(\mathcal{E}_{\Omega_{m}}(0))$ and/or
$\min(\mathcal{E}_{\Omega_{m}\setminus\Omega_{c}}(0))$. This insight is also
applicable to the generic case, where R-Vertices must be weighted. After the
graph partitioning, we denote $\Omega_{\overline{m}}$ to be the robot subset
corresponding to the R-Vertex of the $\overline{m}^{th}$ subgraph. Since
R-Vertices are disconnected from each other, we can follow Corollary 2.1 to
determine the swarm lifetime as

$i\leq\sum_{\overline{m}=1}^{\overline{M}}\omega_{\overline{m}},~{}\omega_{\overline{m}}=\left\lfloor\frac{\min(\mathcal{E}_{\Omega_{\overline{m}}}(0))-i\varepsilon}{pT_{\textsc{c}}+\epsilon}\right\rfloor.$
(26)

In addition, (26) tells us, when the graph partitioning yields multiple
results with the same $\overline{M}$, the one maximizing the upper bound (26)
should be chosen for the sake of lifetime maximization.

###  III-C Optimization in Independent Flat-Fading Channels

Consider the channel quality $(\alpha_{n})$ to be a random variable that is
independent but not necessarily identical with respect to the robot index
$(n)$ and the task index $(i^{\prime})$. Then, the transmit power ($p$) also
varies independently. In this case, the energy consumption model for the
$n^{th}$ robot goes back to the original version (10), and the term
$\zeta_{n}(i)$ can be written into

$\displaystyle\zeta_{n}(i)$
$\displaystyle=T_{\textsc{c}}\sum_{i^{\prime}=0}^{i}\sum_{j\in\Psi_{n}}(p_{n}(i^{\prime}))+|\Psi_{n}|\epsilon,$
(27)
$\displaystyle=w_{n}T_{c}\underbrace{\left(\frac{1}{w_{n}}\sum_{i^{\prime}=0}^{i}\sum_{j\in\Psi_{n}}(p_{n}(i^{\prime}))\right)}_{\triangleq\overline{p}_{n}}+w_{n}\epsilon.$
(28)

Following the discussion leading to Theorem 2 and Corollary 2.1, it is
straightforward to obtain

$w_{n}=\left\lfloor\frac{\mathcal{E}_{n}(0)-i\varepsilon}{\overline{p}_{n}T_{\textsc{c}}+\epsilon}\right\rfloor.$
(29)

In general, the averaged power $\overline{p}_{n}$ is robot dependent. The
lifetime of R-Vertices is limited by the robot with the smallest $w_{m}$.
Hence, the swarm lifetime is given by

$i\leq\sum_{\overline{m}=1}^{\overline{M}}\omega_{\overline{m}},~{}\omega_{\overline{m}}=\min\left\\{\left\lfloor\frac{\mathcal{E}_{n}(0)-i\varepsilon}{\overline{p}_{n}T_{\textsc{c}}+\epsilon}\right\rfloor,n\in\Omega_{\overline{m}}\right\\}.$
(30)

Then, the lifetime maximization is equivalent to maximizing the upper bound of
(30), i.e., following the max-min principle as stated in (12)

$\displaystyle i$
$\displaystyle\leq\max\left(\sum_{\overline{m}=1}^{\overline{M}}\omega_{\overline{m}}\right)$
(31)
$\displaystyle\leq\sum_{\overline{m}=1}^{\overline{M}}\max_{p_{n}(i^{\prime})}\left(\min\left\\{\left\lfloor\frac{\mathcal{E}_{n}(0)-i\varepsilon}{\overline{p}_{n}T_{\textsc{c}}+\epsilon}\right\rfloor,n\in\Omega_{\overline{m}}\right\\}\right).$
(32)

Again, we emphasize that the max-min problem in (32) is NP-hard. Nevertheless,
we can employ the max-min algorithm proposed in [6] to handle the optimization
problem in (32), while the following criteria must be followed in the subset
selection:

1) Criterion 1 in Section II-B specifies the cap of transmit power. Any subset
that does not fulfill Criterion 1 is not chosen for data transmission.

2) When none of R-Vertices satisfy Criterion 1, we choose a subset from those
which are not R-Vertices.

By this means, we incorporate the max-min principle into the LDIP algorithm to
combine the merits of both.

It is worth noting that optimization in fading channels can lead to many
interesting research problems. For instance, current LDIP algorithm randomly
picks up a vertex to be the R-Vertex when there exist multiple candidates of
R-Vertices. This is however too sub-optimum in fading channels where the max-
min principle should be incorporated in the graph partitioning. Moreover,
communication channels can be frequency selective. Therefore, sub-channel
selection mechanism should be incorporated to take the advantage of channel
frequency diversity-gain. Our solution to those research problems will be
presented in the journal paper version of this work.

##  IV Experiment Design and Simulation Results

###  IV-A Model for Generating Robot Subsets

Computer simulations are used to evaluate the proposed subset selection
approach in terms of the lifetime of robot swarm. The major challenge set to
computer simulations is forming robot subsets in Monte Carlo trials. This is
because neither stochastic subset models nor deterministic models are
available in the literature. Deterministic models are relatively easier to
develop; however they are not generic and representative. This is not a big
issue for practical robot swarms since subsets can be formed based on robots’
locations. More specifically, closely located robots have their data strongly
correlated, and they should belong to different subsets. On the other hand,
those very distanced robots have their data weakly correlated, and they can
together form a subset. The MEC server can form subsets based on robots’
location information as well as their empirical data of subset forming.
However, this idea is not readily applicable to computer simulations because
it requires a meaningful and well verified stochastic-geometry model of robot
geo-distribution and location-related data correlations.

As the early root of the multi-robot edge computing research, we propose a
novel and simple stochastic method to generate the robot subsets. Our aim is
to divide $N$ robots into $M$ subsets, with each having at least $K$ robots.
The inequality ($N\leq MK$) must hold as it is a necessary condition for not
excluding any robot in subsets forming. Our subset forming algorithm includes
the following three steps:

Step 1 (Initialization): Set $K$ bins; each bin is given one robot which is
randomly drawn from $N$ robots with an equal probability.

Step 2 (Forming observation groups): For the leftover $(N-K)$ robots, we
randomly throw them into $K$ bins with an equal probability. It is assumed
that robots in the same bin have very correlated data (i.e., correlated
observations); any $K$ robots that are drawn from different bins carry
sufficient information for the edge computing; as specified in Definition 2.

Step 3 (Forming robot subsets): Denote $M_{k}$ the number of robots in the
$k^{th}$ bin. There are three possible cases: a) $M_{k}=M$; b) $M_{k}>M$; c)
$M_{k}<M$.

There is no problem with the case a) since all $M_{k}$ robots can be evenly
allocated to $M$ robot subsets. For the case b), each subset should take at
least $J_{k}\triangleq\lfloor(M_{k})/(M)\rfloor$ robots from the $k^{th}$ bin,
and then we randomly choose $(M_{k}-J_{k}M)$ subsets and allocate one more
robot to each of them. For the case c), we randomly duplicate robots in the
$k^{th}$ bin. For instance, there is a bin accommodating two robots labelled
by $\\{1,2\\}$. Assuming $M=4$, we can randomly generate a set
$\\{1,2,1,1\\}$, $\\{1,2,2,2\\}$, or $\\{1,2,1,2\\}$ for simulation uses
(i.e., making the case $M_{k}=M$).

This stochastic model can cover almost all possible cases of robot subset
forming, including those illustrated in Fig. 1.

###  IV-B Parameter Setting for Simulations

In addition to the forming of robot subsets, there are many other parameters
requiring appropriate setting. Those parameters include:
$\mathcal{E}_{n},\varepsilon,\epsilon,P_{n}$. One of challenging issues is to
determine the relationship between the energy costs for signal transmission,
for modulation and encoding, and for robot task execution, respectively. This
would give us a unified energy consumption model for computer simulations.
Unfortunately, we are not aware of any explicit description of their
relationships in the literature despite a very extensive survey. Therefore, in
our simulations, we assume that a robot’s energy is divided into two fixed and
separated parts, one for communication and the other for task execution. In
order to focus our study in the communication domain, it is assumed that the
bottleneck of robot swarm is the communication energy. In other words, when a
robot has its communication energy used up, this robot counts as ‘dead’.

To simplify the computer simulations, we set $\mathcal{E}_{n}$ and $P_{n}$ to
be identical for all robots and omit the subscript $(\cdot)_{n}$. Moreover, we
ignore the energy cost for modulation and coding (i.e., $\epsilon$) since they
are often negligibly small when comparing to the radio transmission energy.
The term $\mathcal{E}$ is now only referred to the total energy for
communications. In our experiments, it is set to allow a robot conducting
$200$ data transmissions in the AWGN channel with the SNR of $10$ dB. This
essentially sets a robot’s lifetime in the AWGN channel (i.e., $i=200$).

###  IV-C Simulation Results and Discussion

Our computer simulations mainly include two experiments, concerning
communication channels to be AWGN and independent and identically distributed
(i.i.d.) Rayleigh, respectively. We consider two baselines:

  * •

Conventional single-user equivalent edge computing, where all robots send
their data to the MEC server [17]. In the AWGN channel, this baseline approach
straightforwardly gives the swarm lifetime of $i=200$ according to our
setting, and we aim to demonstrate the gain of exploiting source correlation
in the multi-user edge computing.

  * •

Max-min algorithm for subset selection. This algorithm is extended from the
max-min user selection approach in sensor networks [6], but we apply it on the
subset level. Strictly speaking, this is a novel algorithm, and we use it as a
baseline to demonstrate advantages of the proposed LDIP algorithm.

The approach under evaluation is the LDIP-enabled subset selection algorithm
(see (32) for the generic form). The one with only R-Vertices involved in the
subset selection is named LDIP-R-Vertices, and the one with all vertices
involved in the subset selection is named LDIP-All.

#### Experiment 1

The objective of this experiment is to evaluate the proposed approach in the
AWGN channel. We simulate a robot swarm consisting of $(N=30)$ robots. Each
robot subset has at least ($K=8$) robots. Fig. 2 illustrates the swarm
lifetime (averaged over $500$ Monte Carlo trials) as a function of the number
of subsets ($M$). It is shown that subset-selection approaches can improve the
swarm lifetime by $25\%\sim 75\%$. Such a significant gain is due to the
exploitation of data source correlation.

Figure 2: The average swarm lifetime as a function of $M$ in AWGN channels.

It can also be observed that LDIP approaches outperform the max-min approach
in the subset selection. This is because, in the AWGN channel, (32) reduces to
(17), where the round-robin subset selection strategy is optimum (see Theorem
1), and the max-min principle does not offer any additional gain. For the same
reason, LDIP-R-Vertices and LDIP-All show identical performances.

#### Experiment 2

With the same system setup as in Experiment 1, this experiment is interested
in the performance in i.i.d. Rayleigh channels. Experiment 2 is different from
Experiment 1 mainly in two folds: 1) (32) does not reduce to (17), and thus
the max-min principle plays a vital role; 2) power adaptation is needed in
fading channels (see Criterion 1). In our simulations, the transmit power
($p_{n}$) has a power cap, which is set three times of the transmit power in
the AWGN channel. The simulation results are plotted in Fig. 3. It is observed
that the conventional approach has the swarm lifetime significantly reduced.
This is because robots have to pay much more transmit power to combat channel
fades. Compared to the results in AWGN channels, subset-selection approaches
also have the swarm lifetime largely reduced. However, their performances are
significantly better than the conventional approach. Another remarkable
phenomenon is that LDIP-All offers the best performance, and LDIP-R-Vertices
shows worse performance than the max-min approach. This implies that the max-
min principle is vital in fading channels, and all subsets must be considered
in the subset selection. LDIP-All combines the merits of LDIP and max-min
principles, and thus achieves the best performance.

##  V Conclusion and Outlook

In this paper, a novel subset selection concept has been presented for multi-
user edge computing, with the aim to maximize the lifetime of robot swarm
through efficient exploitation of the data source correlation. Major
contribution of this work includes a subset model describing the data source
correlation between robots, an objective function for lifetime optimization,
as well as a graph partitioning-based optimization approach. All of the
proposed models and approaches are novel, and they have demonstrated
remarkable gains both in the AWGN and Rayleigh fading channels.

Moreover, we have highlighted the future direction towards this topic, which
includes the LDIP optimization and user scheduling in frequency-selective
fading channels.

Figure 3: The average swarm lifetime as a function of $M$ in Rayleigh
channels.

## References

  * [1] R. Arnold, K. Carey, B. Abruzzo, and C. Korpela, “What is a robot swarm: A definition for swarming robotics,” in _Proc. IEEE Annu. Ubiquitous Comput. Electron. Mobile Commu. Conf._ , New York, USA, 2019, pp. 0074–0081.
  * [2] T. Qiu, J. Chi, X. Zhou, Z. Ning, M. Atiquzzaman, and D. O. Wu, “Edge computing in industrial internet of things: Architecture, advances and challenges,” _IEEE Commun. Surveys Tuts._ , vol. 22, no. 4, pp. 2462–2488, 4th Quart. 2020.
  * [3] K. Wang, F. Fang, D. B. d. Costa, and Z. Ding, “Sub-channel scheduling, task assignment, and power allocation for OMA-based and NOMA-based MEC systems,” _IEEE Trans. Commun._ , vol. 69, no. 4, pp. 2692–2708, Apr. 2021\.
  * [4] P. Zhao, H. Tian, C. Qin, and G. Nie, “Energy-saving offloading by jointly allocating radio and computational resources for mobile edge computing,” _IEEE Access_ , vol. 5, pp. 11 255–11 268, 2017.
  * [5] S. Zhang, N. Yi, and Y. Ma, “Correlation-based device energy-efficient dynamic multi-task offloading for mobile edge computing,” in _Proc. IEEE Veh. Technol. Conf._ , Helsinki, Finland, 2021, pp. 1–5.
  * [6] S. Gupta and J. Chakareski, “Lifetime maximization in mobile edge computing networks,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 3, pp. 3310–3321, Mar. 2020.
  * [7] B. Nour and S. Cherkaoui, “How far can we go in compute-less networking: Computation correctness and accuracy,” _IEEE Netw._ , vol. 36, no. 4, pp. 197–202, Jul./Aug. 2022.
  * [8] H. El Alami and A. Najid, “ECH: An enhanced clustering hierarchy approach to maximize lifetime of wireless sensor networks,” _IEEE Access_ , vol. 7, pp. 107 142–107 153, 2019.
  * [9] N. Suri, G. Benincasa, R. Lenzi, M. Tortonesi, C. Stefanelli, and L. Sadler, “Exploring value-of-information-based approaches to support effective communications in tactical networks,” _IEEE Commun. Mag._ , vol. 53, no. 10, pp. 39–45, 2015.
  * [10] S. D. Okegbile, B. T. Maharaj, and A. S. Alfa, “A multi-user tasks offloading scheme for integrated edge-fog-cloud computing environments,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 7, pp. 7487–7502, Jul. 2022.
  * [11] C.-F. Liu, M. Bennis, M. Debbah, and H. V. Poor, “Dynamic task offloading and resource allocation for ultra-reliable low-latency edge computing,” _IEEE Trans. Commun._ , vol. 67, no. 6, pp. 4132–4150, Jun. 2019.
  * [12] H. Yetgin, K. T. K. Cheung, M. El-Hajjar, and L. H. Hanzo, “A survey of network lifetime maximization techniques in wireless sensor networks,” _IEEE Commun. Surveys Tuts._ , vol. 19, no. 2, pp. 828–854, 2nd Quart. 2017\.
  * [13] J. G. Proakis and M. Salehi, _Digital communications_. McGraw-hill New York, 2001, vol. 4.
  * [14] J. Hou, Y. Ma, N. Yi, and R. Tafazolli, “Reduced-complexity coordinated beamforming for multicell downlink max–min SINR problem,” _IEEE Wireless Commun. Lett._ , vol. 3, no. 4, pp. 353–356, Aug. 2014.
  * [15] D. B. West, _Introduction to graph theory_ , Upper Saddle River, USA: Prentice Hall, 2001.
  * [16] Z. Wu, H. R. Karimi, and C. Dang, “An approximation algorithm for graph partitioning via deterministic annealing neural network,” _Neural Netw._ , vol. 117, pp. 191–200, 2019.
  * [17] J. Liu and Q. Zhang, “Offloading schemes in mobile edge computing for ultra-reliable low latency communications,” _IEEE Access_ , vol. 6, pp. 12 825–12 837, 2018.


# Resource-Efficient Transfer Learning From Speech Foundation Model Using
Hierarchical Feature Fusion

###### Abstract

Self-supervised pre-training of a speech foundation model, followed by
supervised fine-tuning, has shown impressive quality improvements on automatic
speech recognition (ASR) tasks. Fine-tuning separate foundation models for
many downstream tasks are expensive since the foundation model is usually very
big. Parameter-efficient fine-tuning methods (e.g. adapter, sparse update
methods) offer an alternative paradigm where a small set of parameters are
updated to adapt the foundation model to new tasks. However, these methods
still suffer from a high computational memory cost and slow training speed
because they require backpropagation through the entire neural network at each
step. In the paper, we analyze the performance of features at different layers
of a foundation model on the speech recognition task and propose a novel
hierarchical feature fusion method for resource-efficient transfer learning
from speech foundation models. Experimental results show that the proposed
method can achieve better performance on speech recognition task than existing
algorithms with fewer number of trainable parameters, less computational
memory cost and faster training speed. After combining with Adapters at all
layers, the proposed method can achieve the same performance as fine-tuning
the whole model with $97\%$ fewer trainable encoder parameters and $53\%$
faster training speed.

Index Terms—  speech recognition, foundation model, transfer learning

##  1 Introduction

A foundation model [1] is usually a big model trained on broad data (generally
using self-supervision at scale) that can be fine-tuned to a wide range of
downstream tasks and has aroused extensive attention due to its impressive
quality improvements and emergent capabilities [2, 3, 4, 5]. In speech
community, self-supervised pre-training speech foundation models on a large
amount of unsupervised speech has shown impressive quality improvements on
various speech recognition tasks [6, 7]. There are two main categories of
speech self-supervised learning algorithms. One direction is to reconstruct
(APC [8], MPC [9]) or predict (Wav2vec [10, 11, 12]) the input feature
directly. The other direction is building a BERT-style self-supervised
learning model by bridging the gap between continuous speech signal and
discrete text tokens, such as Wav2vec 2.0 [13], HuBERT [14], w2v-BERT [15] and
BEST-RQ [16]. After pre-training the speech foundation model using the self-
supervised loss, we initialize the encoder of the downstream task using the
pre-trained model and fine-tune it on the supervised data.

A large general-purpose foundation model with millions or even billions of
parameters can be adapted to many downstream tasks. However, it is challenging
to perform separate adaptations for many tasks efficiently with only a small
amount of supervised data each task. There have been existing works
investigating to reduce the number of parameters required for fine-tuning the
foundation model. BitFit [17] proposes a sparse-finetuning method where only
the bias terms of the foundation model are updated. Houlsby et al. [18]
propose to insert Adapter modules between the layers in the fixed pre-trained
model and each module is a small trainable feed-forward neural network. Other
works [19, 20] reduce the number of parameters further by exploiting a low-
rank approximation of the Adapter. Although these parameter-efficient methods
achieve decent performance on the downstream task with a significant reduction
in the trainable parameters, their required computational memory cost and
training time are still very high because of the following two reasons: 1)
using the output of the highest layer in the foundation model only for
downstream tasks, which leads to the inefficiency of the feature usage and
requires to update the foundation model to adapt it to the downstream tasks;
2) adding/updating sparse parameters in the foundation model, which requires a
full backpropagation process from the top to the bottom of the network to
compute the gradients of the trainable parameters. Thus, a resource-efficient
transfer learning method, which can achieve comparable performance with small
number of trainable parameters, low computational memory cost and fast
training speed, is required for efficient adaptation of the foundation model
to many downstream tasks.

Recently, Pasad et al. [21] analyze the layer-wise features of a self-
supervised (wav2vec2.0) pre-trained speech representation model and finds that
the middle layers encode the most contextual and high-level information. The
bottom or top few layers, on the other hand, focus on the lower-level
information and encode more local representations. Arunkumar et al. [22]
investigate the ensemble features of self-supervised pre-trained models for
ASR and finds that features from different self-supervised learning methods
are complementary and the ensemble of features is beneficial for the
downstream speech recognition tasks. Although behaviors of the layer-wise
features and features from multiple self-supervised pre-trained models are
explored, neither of them consider the resource efficiency in the fine-tuning
stage and there is no investigation about the feature fusion of layer-wise
features from a single pre-trained model on downstream tasks.

In this paper, we propose a novel resource-efficient transfer learning method
for speech foundation models. specifically, we treat the foundation model as a
frozen feature extractor and fuse the multi-level features from the foundation
model hierarchically. We conduct extensive experiments to investigate
different ways of feature fusion for the foundation model. Experimental
results show that the proposed method can achieve better performance on the
ASR task than existing parameter-efficient fine-tuning algorithms with fewer
number of trainable parameters, less computational memory cost and faster
training speed. After combining with Adapters at all layers, the proposed
method can achieve the same performance as fine-tuning the whole model with
$97\%$ fewer trainable encoder parameters and $53\%$ faster training speed.

##  2 Experimental Setup

###  2.1 Foundation Model And Task

The foundation model used in the paper is a $2$-layer convolutional network
followed by a $24$-layer conformer encoder with hidden dimension $1024$ and
about $600$M parameters in total. Each conformer layer [23] is a convolution-
augmented transformer network, which consists of attention, feed-forward and
convolutional modules. The model input is a vector of size $128$ logMel
features and SpecAugment [24] is also applied to increase model robustness. We
pre-train the $600$M conformer encoder using the BEST-RQ [16] algorithm for
$800$K steps. For the downstream speech recognition task, we initialize the
encoder using the pre-trained speech foundation model and the output of the
encoder is used as input to an RNN-T [6] along with a $6$-layer LSTM decoder
and dimension $768$. We train with Adam optimizer for both pre-training and
fine-tuning, and use exponential moving averaging (EMA) with decay rate 0.9999
for fine-tuning only. We update the trainable encoder parameters and LSTM
decoder which has $124$M trainable parameters on Voice Search data for $100$K
steps. If not described explicitly, the parameter efficiency refers to the
reduction of the trainable parameters in the encoder only. All experiments are
performed on TPUs.

Fig. 1: Voice Search WER when extracting features from different layers of the
foundation model.

###  2.2 Training Data

We use two sources of training data in this work. Following [6], we collect
$800$K hours unsupervised English Youtube data and pre-train the $600$M
foundation model on the randomly segmented audio-only Youtube speech using the
BEST-RQ algorithm [16]. In addition, the supervised English Voice Search (VS)
data contains $5$K hours of labeled voice search audio [25] and is used to
fine-tune the conformer encoder and RNNT-T decoder for the ASR task. All data
are collected and deidentified in accordance with Google AI principles [26].

###  2.3 Evaluation

In this paper, we calculate the word error rate (WER) on the Voice Search (VS)
test dataset to measure the quality of the model on the downstream speech
recognition task. Apart from WER, we compare the number of trainable
parameters, computational memory cost and training speed at the same time for
resource efficiency. The target of this paper is to propose a method, which
can achieve low WER with small number of trainable parameters, low
computational memory cost and fast training speed.

##  3 Linear Feature Fusion of The Foundation Model

Fig. 2: Linear feature fusion from multiple layers of the foundation model,
using a $6$-layer conformer encoder as an example.

Previous parameter-efficient fine-tuning methods update the sparse parameters
in the foundation model and use the output of the highest encoder layer only
as the input to the RNN-T decoder, while the outputs of the intermediate
layers are dropped after the forward pass. The proposed feature fusion method
treat the foundation model as a frozen feature extractor and fuse the multi-
level features from different layers linearly or hierarchically. Because there
is no need to perform backward pass in the foundation model and only a few
parameters are added on top of the outputs of the intermediate layers, the
proposed feature fusion method is parameter-efficient and computation-
efficient.

###  3.1 Performance of Single Layer Features

To study the performance of the features from different layers of the
foundation model, we extract outputs from layers
$\\{1,3,5,10,12,14,19,21,23\\}$ respectively and update the $124$M 6-layer
LSTM decoder only on the Voice Search data. Figure 1 shows the WER of the
corresponding layers and results present that models using features from
middle layers perform better on the speech recognition task than features from
bottom or top layers. This observation is consistent with [21] that middle
layers encode more contextual and high-level information which is more helpful
for the speech recognition task than bottom or top layers.

Fig. 3: Balanced and unbalanced hierarchical feature fusion methods of the
foundation model, using $6$-layer conformer encoder as an example. FP denotes
a $1$-layer fully-connected network.

###  3.2 Linear Feature Fusion From Multiple Layers

From Section 3.1, we know that features from different layers show different
performance on the downstream speech recognition task. To investigate whether
these features are complementary, we propose a linear feature fusion method
and combine features from different layers linearly. As in Figure 2, we
firstly concatenate the features from multiple layers and project the
concatenated feature to the required dimension using a fully-connected neural
network. The decoder receives the output of the projector as input. All the
conformer layers in the encoder are fixed while we update feature projector
and decoder only using the RNN-T loss.

Table 1: Fusing features from multiple layers of the foundation model. Feature projector is a $1$-layer fully-connected network for all combinations.  Layer index | # Parameters | VS WER
---|---|---
| In Feature Projector |  ($\%$)
$11$ |  $0.6$ M | $11.2$
$23$ |  $0.6$ M | $91.9$
$11,23$ |  $1.3$ M | $10.8$
$5,11,17,23$ |  $2.6$ M | $9.3$
$2,5,8,11,14$ |  $5.2$ M | $8.1$
$17,20,23$ |  |
$1,3,5,7,9,11,13$ |  $7.9$ M | 8.0
$15,17,19,21,23$ |  |
Fig. 4: $\ell_{2}$ norm of the learned weight of each layer when fusing
features from $12$ layers.

We compare the Voice Search (VS) WER when fusing features from
$\\{1,2,4,8,12\\}$ layers. Results in Table 1 demonstrate the benefit of
feature fusion from multiple layers. When fusing features from $12$ layers, we
obtain the best VS WER $8.0\%$ with additional $7.9$M parameters in the
feature projector. Figure 4 shows the norm of the learned weights for each
layer when fusing $12$ features. The figure presents a higher weight for the
middle layers and a lower weight for bottom or top layers. The results
demonstrate that features from middle layers contribute more to the speech
recognition task and adding features from bottom or top layers is also
helpful.

###  3.3 Increasing Depth of The Feature Projector

We also explore to learn non-linear feature fusion by increasing the depth of
the feature projector in Figure 2. In Table 2, we increase the depth of the
fully-connected network from $1$ to $4$ layers with ReLU activation while
extracting features from the same $12$ layers used in the previous
experiments, which was found to give the best results. Experimental results
show that the model gets a better WER with a deeper feature projector and the
VS WER becomes saturated at about $7.4\%$ after adding up to $3$ fully-
connected layers.

Table 2: Increasing depth of the feature projector. Fusing features from $12$ layers as it gives the best results.  # Layers | # Parameters | VS WER
---|---|---
| In Feature Projector |  ($\%$)
$1$ |  $7.9$ M | $8.0$
$2$ |  $8.3$ M | $7.5$
$3$ |  $8.7$ M | 7.4
$4$ |  $9.1$ M | 7.4

##  4 Hierarchical Feature Fusion of the Foundation Model

Knowing that features from different layers encode different levels of
information, we also explore to fuse features in a hierarchical way rather
than linearly. In this section, we propose a hierarchical feature fusion
method and compare it with other parameter-efficient fine-tuning algorithms.

Table 3: Comparison with baselines and prameter-efficient methods. $\downarrow$ denotes the smaller the better. Second column shows the number of trainable parameters in the encoder only for the corresponding compared method and the whole $124$M LSTM decoder are trainable as well. Methods | # Trainable | Computational | Training Speed | VS WER
---|---|---|---|---
| Encoder Params $\downarrow$ | Memory Cost $\downarrow$ | Examples/Sec $\uparrow$ | ($\%)\downarrow$
Fine-tune all |  $606.6$ M |  $13567$ MB | $1270$ | $5.5$
Fine-tune the highest encoder layer (FTHS) |  $25.4$ M |  $7563$ MB | $3616$ | $15.8$
BitFit |  $0.1$ M |  $12443$ MB | $2824$ | $6.5$
Adapter(d=$128$) at all layers |  $6.4$ M |  $12411$ MB | $2810$ | $6.4$
Adapter(d=$256$) at all layers |  $13.3$ M |  $12455$ MB | $2802$ | $6.1$
Adapter(d=$512$) at all layers |  $25.9$ M |  $12486$ MB | $2788$ | $6.1$
Adapter(d=$128$) at layers |  |  |  |
$\\{13,15,17,19,21,23\\}$ |  $2.3$ M |  $9340$ MB | $3251$ | $7.9$
Linear Feature Fusion |  $8.7$ M |  $7573$ MB | $3610$ | $7.4$
HFF-b |  $12.3$ M |  $7648$ MB | $3655$ | $7.0$
HFF-b + Adapter(d=$128$) at layers |  |  |  |
$\\{13,15,17,19,21,23\\}$ |  $13.9$ M |  9653 MB | 3213 | 6.0
HFF-b + Adapter(d=$128$) at all layers |  $18.6$ M |  $12378$ MB | 2750 | 5.5

###  4.1 Hierarchical Feature Fusion From Multiple Layers

As in Figure 3, we compare two hierarchical feature fusion methods (balanced
and unbalanced) for the speech foundation model. For the balanced feature
fusion method (HFF-b), we project and concatenate the neighboring pair-wise
features, treating all layers equally. For the unbalanced feature fusion
method (HFF-ub), on the other hand, we project and concatenate the neighboring
features from bottom to the middle and from top to the middle. The intuition
is that the middle layers encode high-level information while the bottom or
top layers encode low-level information, such that more encoding is required
for the features from these layers.

Table 4: Comparison between balanced and unbalanced hierarchical feature fusion methods. Fusing features from $12$ layers.  Methods | # Parameters | VS WER
---|---|---
| In Feature Projector |
HFF-b |  $12.3$ M | 7.0
HFF-ub |  $12.3$ M | $7.2$

We use a $1$-layer fully-connected network as FP in Figure 3 and the projector
in the “Concat $\&$ Project” is a $3$-layer fully-connected network. The FP
projects a $1024$-d feature to $512$-d, such that the feature dimension
remains unchanged after concatenation. Table 4 shows that both methods achieve
better VS WER than linear feature fusion, and HFF-b performs better than the
HFF-ub on the speech recognition task with the same amount of parameters in
the feature projector. Therefore, we use balanced hierarchical feature fusion
(HFF-b) in the following experiments.

###  4.2 Comparison with Parameter-Efficient Fine-Tuning Methods

To validate the proposed hierarchical feature fusion method, we compare it to
several related algorithms. Specifically, we compare with two representative
and strong parameter-efficient methods: BitFit [27] and Adapter [18]. Each
adapter module is inserted after each conformer encoder layer and is a
randomly initialized $2$-layer feed-forward network with the bottleneck
dimension $d$ from $\\{128,256,512\\}$. We also fine-tune the highest
conformer encoder layer (FTHST) as a baseline, which is computation-efficient
because no backpropagation is required for the lower encoder layers. The
parameter-efficient methods are applied to fine-tune the $600$M conformer
encoder only, and the whole randomly initialized $124$M LSTM decoder is also
updated simultaneously. Because the LSTM decoders are the same for all
compared methods, we only compare the number of trainable encoder parameters
in the experiments regarding parameter efficiency. Although the best VS WER
can be achieved if we fine-tune all parameters of the model, it costs too much
computational memory $13567$MB and the training speed is very slow at $1270$
examples/sec. Results in Table 3 show that Adapter’s performance is better
than BitFit or FTHS, but gets stuck at $6.1$ VS WER even if increasing the
bottleneck dimension from $128$ to $512$. However, the Adapter(d=128) at all
layers’s training speed is $22\%$ slower and computational memory cost is
$64\%$ higher than FTHS. With a very similar computational memory cost and
training speed to FTHS, HFF-b can improve VS WER from $15.8\%$ to $7.0\%$.
Comparing with Adapter(d=128) at layers $\\{13,15,17,19,21,23\\}$, HFF-b
achieves better VS WER with $12\%$ faster training speed and $18\%$ lower
computation memory cost. Combining the HFF-b with Adapter(d=128) at layers
$\\{13,15,17,19,21,23\\}$, we can achieve better VS WER $6.0\%$ than all
compared parameter-efficient methods with fewer number of trainable
parameters, less computational memory cost and faster training speed. If
combining the proposed HFF-b with Adapter($d=128$) at all layers, we can
achieve the same WER as fine-tuning all parameters of the RNN-T model with
$97\%$ fewer trainable encoder parameters and $53\%$ faster training speed.

##  5 Conclusion

In this paper, we analyze the behavior of features from different layers of
the foundation model for speech recognition task and propose a hierarchical
feature fusion method for resource-efficient transfer learning from the speech
foundation model. Extensive results demonstrate that it achieves promising
performance on the speech recognition task with fewer trainable encoder
parameters, less computational cost and faster training speed.

## References

  * [1] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al.,  “On the opportunities and risks of foundation models,”  arXiv preprint arXiv:2108.07258, 2021.
  * [2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.,  “Language models are few-shot learners,”  Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * [3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763.
  * [4] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, et al.,  “Exploring the limits of transfer learning with a unified text-to-text transformer.,”  J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020.
  * [5] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov,  “Roberta: A robustly optimized bert pretraining approach,”  arXiv preprint arXiv:1907.11692, 2019.
  * [6] Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang, et al.,  “Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, 2022.
  * [7] D. Hwang, A. Misra, Z. Huo, N. Siddhartha, S. Garg, D. Qiu, K. C. Sim, T. Strohman, F. Beaufays, and Y. He,  “Large-scale asr domain adaptation using self-and semi-supervised learning,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6627–6631.
  * [8] Y.-A. Chung and J. Glass,  “Generative pre-training for speech with autoregressive predictive coding,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 3497–3501.
  * [9] W. Wang, Q. Tang, and K. Livescu,  “Unsupervised pre-training of bidirectional speech encoders via masked reconstruction,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6889–6893.
  * [10] A. v. d. Oord, Y. Li, and O. Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv preprint arXiv:1807.03748, 2018.
  * [11] S. Schneider, A. Baevski, R. Collobert, and M. Auli,  “wav2vec: Unsupervised Pre-Training for Speech Recognition,”  in Proc. Interspeech 2019, 2019, pp. 3465–3469.
  * [12] Z. Huo, D. Hwang, K. C. Sim, S. Garg, A. Misra, N. Siddhartha, T. Strohman, and F. Beaufays,  “Incremental layer-wise self-supervised learning for efficient speech domain adaptation on device,”  arXiv preprint arXiv:2110.00155, 2021.
  * [13] A. Baevski, H. Zhou, A. Mohamed, and M. Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  arXiv preprint arXiv:2006.11477, 2020.
  * [14] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [15] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu,  “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 244–250.
  * [16] C.-C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu,  “Self-supervised learning with random-projection quantizer for speech recognition,”  arXiv preprint arXiv:2202.01855, 2022.
  * [17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [18] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly,  “Parameter-efficient transfer learning for nlp,”  in International Conference on Machine Learning. PMLR, 2019, pp. 2790–2799.
  * [19] R. Karimi Mahabadi, J. Henderson, and S. Ruder,  “Compacter: Efficient low-rank hypercomplex adapter layers,”  Advances in Neural Information Processing Systems, vol. 34, pp. 1022–1035, 2021.
  * [20] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen,  “Lora: Low-rank adaptation of large language models,”  arXiv preprint arXiv:2106.09685, 2021.
  * [21] A. Pasad, J.-C. Chou, and K. Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 914–921.
  * [22] A. Arunkumar, V. N. Sukhadia, and S. Umesh,  “Investigation of ensemble features of self-supervised pretrained models for automatic speech recognition,”  arXiv preprint arXiv:2206.05518, 2022.
  * [23] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [24] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le,  “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,”  in Proc. Interspeech 2019, 2019, pp. 2613–2617.
  * [25] A. Narayanan, A. Misra, K. C. Sim, G. Pundak, A. Tripathi, M. Elfeky, P. Haghani, T. Strohman, and M. Bacchiani,  “Toward domain-invariant speech recognition via large scale training,”  in 2018 IEEE Spoken Language Technology Workshop (SLT), 2018, pp. 441–447.
  * [26] “Artificial intelligence at google: Our principles,” https://ai.google/principles/.
  * [27] E. B. Zaken, S. Ravfogel, and Y. Goldberg,  “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,”  arXiv preprint arXiv:2106.10199, 2021.