# Resource Allocation of Federated Learning for the Metaverse with Mobile
Augmented Reality

Xinyu Zhou  Chang Liu  Jun Zhao A shorter version of this work containing
fewer results has been accepted to 2022 IEEE 42nd International Conference on
Distributed Computing Systems (ICDCS) [1]. Improvements over [1] are
explicitly discussed in this paper.The authors are all with Nanyang
Technological University, Singapore. Contact<EMAIL_ADDRESS>
(December 2020)

###### Abstract

The Metaverse has received much attention recently. Metaverse applications via
mobile augmented reality (MAR) require rapid and accurate object detection to
mix digital data with the real world. Federated learning (FL) is an intriguing
distributed machine learning approach due to its privacy-preserving
characteristics. Due to privacy concerns and the limited computation resources
on mobile devices, we incorporate FL into MAR systems of the Metaverse to
train a model cooperatively. Besides, to balance the trade-off between energy,
execution latency and model accuracy, thereby accommodating different demands
and application scenarios, we formulate an optimization problem to minimize a
weighted combination of total energy consumption, completion time and model
accuracy. Through decomposing the non-convex optimization problem into two
subproblems, we devise a resource allocation algorithm to determine the
bandwidth allocation, transmission power, CPU frequency and video frame
resolution for each participating device. We further present the convergence
analysis and computational complexity of the proposed algorithm. Numerical
results show that our proposed algorithm has better performance (in terms of
energy consumption, completion time and model accuracy) under different weight
parameters compared to existing benchmarks.

###### Index Terms:

Metaverse, mobile augmented reality, federated learning, FDMA, resource
allocation.

##  I Introduction

Nowadays, we can develop intelligent machine learning models that reliably
recognize and classify complicated things in the real world due to advances in
deep learning. This breakthrough has motivated the advancement of Augmented
Reality (AR) technology, which improves the human perception of the world by
merging the actual world with the virtual space. Mobile Augmented Reality
(MAR) applications operate on mobile platforms and allow users to access
services through their devices’ interfaces. MAR and augmented reality (AR)
further empower the Metaverse, which is considered a next evolution of the
Internet [2]. By implementing machine learning technologies on devices, more
intelligent MAR applications are developed in different Metaverse scenarios
such as entertainment, communications and medicine.

Imagine a Metaverse scenario where putting on a pair of glasses, you can see
descriptions of buildings, which is convenient to find the destination during
traveling. Thus, in MAR for the Metaverse, rapid and accurate object
recognition is critical in bringing digital data together with environmental
aspects. This scenario can be addressed by different approaches. For example,
mobile devices can upload their images to a proximate edge server with a
strong computing power to run deep learning algorithms to save computational
latency and energy [3, 4].

In this paper, we assume each AR user of the Metaverse can generate data for
object detection tasks. For example, users can mark labels for images
generated by AR devices and those images are used for training afterwards.  To
create an object detection model with high accuracy, usually we need to feed
it with a large size of training data. Due to limitations of user privacy,
data security and competition among industries, usually, the data is limited.
Besides, the concern around data privacy is rising. Related laws and
regulations are appearing, such as the General Data Protection Regulation
(GDPR) of the European Union (EU), so it may be difficult to collect sensitive
data together to learn a model. Mobile devices offloading their training tasks
to a cloud/edge server may raise privacy concerns.  Training a model solely on
one device could be a solution, but the amount of data collected by one device
may not be enough for training a good model. We then consider incorporating
federated learning (FL) into the design of MAR systems for the Metaverse.

FL can collaboratively train a machine learning model while preserving data on
its local device distributedly. It is a special distributed learning framework
proposed by Google in [5]. In the FL algorithm of [5], each device trains a
model locally for a certain number of local iterations and periodically
uploads its parameters to a server to build a shared global model. By
incorporating FL, on the one hand, we can utilize the computation resources of
each device. On the other hand, we do not need to upload video frames to a
central server for training, which protects user privacy. We call such a
system as FL-MAR system.

Motivation. The hardware performance is improving continuously according to
Moore’s Law [6], and thus the computational resources of mobile devices will
most likely become abundant in the future. Therefore, it is worth
investigating how to run machine learning experiments on mobile devices. One
MAR device usually has limited data, which may not be enough to train a good
model solely. That is why we are considering integrating FL into MAR
applications. Each MAR device can benefit from joining FL. FL only needs each
device to upload its parameters to a server, and the server will send back the
average parameter to each device. Thus, each device can improve its model
without sharing its data, which protects user privacy. Hence, it is worthwhile
to study the application of FL to MAR for building intelligence into the
Metaverse.

Challenges. Applying FL to MAR applications for the Metaverse still has high
requirements of device memory, computational capability and communication
bandwidth. There are several challenges to the deployment of FL to MAR
applications: (1) Limited bandwidth results in long latency between clients
and the server and, consequently, affects the convergence time of the global
model. (2) A large amount of energy is needed because a satisfactory model
requires quite a few local computation and communication rounds. (3) MAR
devices may be in a low-battery condition while training. How to utilize its
computational resources (e.g., computational frequency) and adjust video frame
resolutions during training to save energy and time? Besides, finding optimal
resource (e.g., transmission power, bandwidth, CPU frequency, etc.) allocation
strategies can save both communication and computation resources [7, 8, 9]. An
optimal resource allocation strategy will allow us to reduce the execution
latency and energy consumption and accelerate the convergence of the global
model. Therefore, it is essential to consider adjusting bandwidth,
transmission power and CPU clock frequency for each participating device to
optimize energy consumption and completion time [10]. Moreover, different
scenarios have different energy and latency demands. For example, connected
autonomous vehicles in an intelligent transportation system require low-
latency control decisions, while personal computers may tolerate higher
latency to utilize the idle time to train their model. Besides, the object
recognition accuracy is related to the video frame resolution of MAR devices
[11]. Higher resolution means higher computational latency and energy
consumption. Thus, it raises the following question: how to assign the
transmit power, bandwidth, CPU frequency and video frame resolution to each
user device to find the optimal trade-off between energy consumption, the
total completion time and model accuracy of the FL-MAR system while
accommodating different scenarios?

Motivated by the above question, we consider a basic FL-MAR model for the
Metaverse with frequency-division multiple access (FDMA) system and formulate
a joint optimization of energy, time and accuracy problem over the wireless
network. The objective function includes a weighted combination of total
energy consumption, completion time and model accuracy. The participating
devices in FL are under power constraints and bandwidth constraints while
pursuing as little energy, time consumption and much accuracy as possible.

Comparison with our ICDCS paper [1]. We extend the algorithm in [1] to an MAR
scenario for the Metaverse. In [1], we do not consider any optimization
variable or objective function related to MAR. The optimization problem in [1]
is to minimize the total energy and time consumption. In this work, we add one
more optimization variable—video frame resolution in the context of MAR for
the Metaverse. The objective is to optimize energy, time, and model accuracy
simultaneously. To evaluate the impact of video frame resolution, object
detection experiments are implemented on COCO dataset [12] using modified
YOLOv5m [13] model.

Contributions. In summary, the main contributions of this paper are listed as
follows:

  * •

To the best of our knowledge, we are the first to incorporate FL into the
energy-efficient MAR system design for the Metaverse. To utilize the
communication and computational resources efficiently, we formulate an
optimization problem with variables including transmission power, bandwidth,
CPU frequency and video frame resolution, with the objective function
incorporating energy, time, and model accuracy.

  * •

One part of the optimization problem includes a non-convex sum-of-ratio form
of communication energy consumption, which cannot be solved by traditional
optimization techniques. To tackle this problem, we propose to decompose the
original problem into two subproblems: the first subproblem is to minimize
computation energy, completion time and model accuracy, and the second is to
minimize the communication energy. The first one is solved through its
Lagrange dual problem. The second subproblem is a sum-of-ratio minimization
problem, which is transformed into a convex subtractive-form problem through a
Newton-like method similar to [14] used for solving a general fractional
programming problem.

  * •

Convergence analysis and time complexity are provided. Numerical results show:
1) Under different weight parameters (i.e., different requirements), our
resource allocation algorithm can find a corresponding solution to jointly
minimize energy consumption and completion time and maximize the model
accuracy. 2) Our joint optimization framework’s performance is better than the
communication-only optimization and computation-only optimization frameworks.
3) Our resource allocation scheme outperforms the benchmarks, especially
regarding total energy consumption.

In Section II, we discuss the related work. In Section III, we introduce the
system model and necessary notations. Problem formulation is provided in
Section IV. We elaborate the proposed algorithm in Section V. Then Section VI
presents the algorithm’s convergence and computational complexity. Numerical
results are shown in Section VII. Finally, Section VIII concludes the paper.

##  II Related Work

In this section, we survey work related to our study in terms of MAR, the
Metaverse and FL.

Related studies on MAR and Metaverse. Recently, MAR systems and their
empowerment for the Metaverse are attracting more attention thanks to the
improvements in wireless communications and end devices’ computing capacity
[15, 16]. The MAR applications, especially for the Metaverse, are CPU-
intensive and memory-intensive since they perform various heavy computational
tasks.

A few studies focus on developing new methods to offload tasks to the cloud or
edge server to reduce energy consumption and improve the performance of MAR
systems. For example, [17] proposes an edge-assisted scheme to schedule
offloading requests across multiple applications. [18] designs an edge-based
MAR system and proposes an optimization algorithm to change MAR clients’
configuration parameters and guide edge server radio resource allocation
dynamically to save energy consumption and latency while maximizing the model
accuracy. [11] devises an edge orchestrator to complete object analysis for
MAR. [19] allocates computation resources dynamically and utilizes computation
resources on the edge server.

Another genre of work aims at studying resource allocation schemes in MAR
systems with mobile edge computing [20, 21, 22, 23]. For example, [20] studies
the joint problem of viewport rendering offloading and resource allocation for
wireless VR streaming. VideoStorm [21] jointly optimizes resources and quality
in the video analytics system. [22] designs an algorithm to minimize the
maximum communication and computation latency among all VR users under the
limited computation resource and transmission power. [23] proposes a joint
resolution and power control scheme considering energy efficiency, latency and
accuracy trade-off in the MAR system.

There are also studies devising new on-device deep learning frameworks in MAR
[24, 25, 26, 27]. However, an individual data holder may not have sufficient
local data or features for training an effective model. In our work, we
incorporate the framework of federated learning, where mobile devices are
allowed to learn a shared global model collaboratively.

Related work on the Metaverse has been growing recently [28, 29, 30, 31]. For
instance, [28] and [29] are about the application of FL to the industrial
Metaverse. [30] stands in the perspective of economics to assess and improve
VR services for the Metaverse. [31] studies the resource allocation problem in
a Metaverse MAR service model. However, there still lacks the combination of
FL and MAR in the Metaverse scenario.

Related studies on FL. In FL, due to limited communication resources and a
large number of user devices, the convergence rate of the global model may be
slow, resulting in high latency and communication costs. Moreover, some
devices might be offline during the training process, which will hamper the
model.

From the perspective of wireless communications, related studies pay much
attention to the whole FL system and devise an approach to allocating energy
resources to each device in the system. The resources related to energy
consumption involve limited bandwidth, transmit power, local device’s CPU
frequency and so forth. Considering all devices joining each communication
round via FDMA, a joint learning and communication problem is formulated in
[7]. Then, the authors investigate how to allocate time, bandwidth, CPU
frequency and learning accuracy to each device under a latency constraint.
Similarly, instead of considering FDMA, Dinh _et al._ [9] come up with an FL
algorithm via time-division multiple access (TDMA) called FEDL and study the
trade-off between the convergence time and energy consumption of FEDL. Some
papers also study how to minimize energy consumption in hierarchical FL [8,
32]. [8] proposes a resource allocation and edge association scheme in a
cloud-edge-client federated learning framework and shows it to converge to a
stable system point. Resource allocation and edge-device selection policy are
discussed in [32], which shows that the proposed strategy outperforms the
uniform bandwidth allocation and all-device-participating strategy.

Related work on FL with AR. We only found one article [33] that incorporates
AR into the FL study. To obtain an accurate global model and reduce the
computational and power costs for AR users, [33] proposed to use an FL-based
mobile edge computing paradigm to solve object recognition and classification
problem. However, [33] only proposed an FL framework without considering how
to allocate communication and computation resources in this framework.

Novelty of our work. The previously mentioned approaches, which use remote
clouds to ingest images or video streams, are still time-consuming and energy-
intensive. Besides, offloading tasks to cloud servers may incur privacy
leakage. In contrast, performing tasks locally ensures a faster experience and
provides more secure privacy protection. By incorporating the framework of FL,
we allow performing model training locally on devices and periodically
uploading their parameters to a base station to create a shared global model.
Without assuming that devices and servers have sufficient bandwidth, we
investigate a more practical setting where the bandwidth between devices and
servers is limited. To reduce energy consumption and latency, we propose an
energy-efficient MAR system for the Metaverse, where the weighted combination
of energy consumption and total completion time is optimized. In addition to
service latency and energy consumption, learning accuracy is equally vital to
Metaverse MAR systems. Thus, balancing service latency, energy consumption and
learning accuracy is essential. Although the above studies [7, 8, 9] consider
latency and energy consumption, they do not take machine learning accuracy
into account. [11] investigates the trade-off between the latency and
analytics accuracy in edge-based MAR systems and develops an algorithm to
improve the system performance, but it fails to consider the energy
consumption of MAR devices. Although [18] considers optimizing the video frame
resolution, CPU frequency and the bandwidth of each user, its scenario is
indoor without incorporating FL and it defines the transmission rate by using
CPU frequency which is in a different context from ours. Moreover, [31]
optimizes the resource allocation for each user in a Metaverse MAR system, but
it does not incorporate FL. In our work, we maximize the accuracy for each
local device while achieving low latency and energy consumption at the same
time under a wireless communication scenario.

Figure 1: Federated learning with mobile augmented reality (FL-MAR) for the
Metaverse.

##  III System Model

This section presents the Metaverse system model of FL with MAR over wireless
communications. We consider a star network with a base station, as shown in
Fig. 1. The Metaverse service is provided through the base station. The base
station serves $N$ MAR devices from $1$ to $N$, and the set of the $N$ MAR
devices is represented by $\mathcal{N}:=\\{1,2,\ldots,N\\}$. Let
$n\in\mathcal{N}$ denote the device index. The $n$th device contains a dataset
with $D_{n}$ samples.

Then, the overall FL model is to solve the problem
$\textstyle{\min_{\bm{w}\in\mathbb{R}^{p}}\ell(\bm{w})=\sum_{n=1}^{N}\frac{D_{n}}{D}\ell_{n}(\bm{w})}$,
where $D=\sum_{n=1}^{N}D_{n}$ and $\bm{w}$ is the global model parameter.

As shown in Fig. 1, the FL process is iterative and consists of several global
rounds. At the beginning of each global round, each device has the same model
parameter broadcasted by the base station at the end of the previous global
round. Then each device performs training on its local dataset for several
local iterations and obtains an updated model parameter value. At the end of
this global round, all devices upload their model parameters to the base
station to compute a weighted average of the model parameter. Then the base
station broadcasts the new parameter value to all devices to start the next
global round. Such an uploading and broadcasting process is called global
communication.

Between two global aggregations, each device $n$ trains its own model for a
few local iterations, and the aim is to minimize the loss function defined at
device $n$ as
$\textstyle{\ell_{n}(\bm{w})=\frac{1}{D_{n}}\sum_{i=1}^{D_{n}}\ell_{n,i}(\bm{w})}.$
The function $\ell_{n,i}(\bm{w})$ measures the error between the model
prediction and the ground truth label, and $i$ represents the $i$th sample on
device $n$. We consider that in each local iteration, device $n$ uses all of
its $D_{n}$ data samples, not just a subset (future work can consider using
just a subset).

In this paper, we formulate a joint optimization of energy, completion time
and model accuracy problem in FL via Frequency Division Multiple Access
(FDMA). FDMA means the spectral resources are divided among MAR devices so
that the communication between one MAR device and the base station has no
interference with the communication between any other MAR device and the base
station. The objective function is a weighted combination of total energy,
time, and model accuracy. Total energy consumption, total completion time and
model accuracy model are discussed in Sections III-A, III-B and III-C,
respectively.

Notations in this paper are listed in Table II in Appendix A. Bold symbols
stand for vectors.

###  III-A Total Energy Consumption

The total energy consumption, $\mathcal{E}$, comprises wireless transmission
energy consumption and local computation energy consumption. The same as many
studies [9, 8, 7] in the literature, we do not consider the energy consumption
of the base station (i.e., the central server).

####  III-A1 Transmission energy consumption

Generally, transmission time consists of the uplink and downlink time.
However, since the output power of the base station is significantly larger
than a user device’s maximum uplink transmission power, the downlink time is
much smaller than the uplink time. Thus, we neglect the downlink time and only
consider the uplink time, as in [9, 7, 8].

Following Shannon’s pioneer work [34], we express the data transmission rate,
$r_{n}$, as

$\displaystyle\textstyle{r_{n}=B_{n}\log_{2}(1+\frac{g_{n}p_{n}}{N_{0}B_{n}})},$
(1)

where $B_{n}$ is the bandwidth allocated to user $n$, $N_{0}$ is the power
spectral density of Gaussian noise, $g_{n}$ is the channel gain between user
device $n$ and the base station, and $p_{n}$ is the transmission power. Note
that there is no interference in FDMA since the channels assigned to users are
separated by different frequencies. The data transmission will happen in each
global aggregation. If the transmission data size of user device $n$ is
$d_{n}$ (bits), the uplink transmission time from user device $n$ to the base
station is

$\displaystyle\textstyle{T^{up}_{n}=d_{n}/r_{n}},$ (2)

The transmission energy consumption of user device $n$ is

$\displaystyle\textstyle{E^{trans}_{n}=p_{n}T^{up}_{n}}.$ (3)

####  III-A2 Local computation energy consumption

Following [35], and assuming that the CPU frequency $f_{n}$ (i.e., the number
of CPU cycles per second) of device $n$ does not change once it is decided at
the beginning of the FL process, we express the energy consumed in one local
iteration of user device $n$ as:

$\displaystyle\kappa c_{n}D_{n}f_{n}^{2},$ (4)

where $\kappa$ is the effective switched capacitance, and $c_{n}$ is the
number of CPU cycles per standard sample on device $n$. By a standard sample,
we mean a square video frame with a resolution of $s_{\text{standard}}\times
s_{\text{standard}}$ pixels.

However, each MAR device may not always use the exact video frame resolution
to train its local model. For instance, if a user forgets to charge the VR
device, the device cannot use high-resolution video frames to train its model
under such a low-battery condition. Intuitively, processing higher-resolution
video frames require more CPU cycles, while lower-resolution video frames need
less. Thus, the calculation of CPU cycles $c_{n}D_{n}$ in Eq. (4) cannot stand
for the cycles needed for high-resolution video frames nor low-resolution
video frames.

We now analyze the needed CPU cycles for different MAR video frame
resolutions. For simplicity, our paper considers only square video frames
(i.e., images), which means that the height equals the width. On device $n$,
let the video frame resolution be $s_{n}\times s_{n}$ pixels. Sometimes we
abuse the description and refer to $s_{n}$ as the video frame resolution for
simplicity.

We next focus on how to quantify the CPU cycles needed for video frames with
different resolutions. For a video frame with a resolution of $s_{n}^{2}$
pixels and $s_{n}>s_{\text{standard}}$, it needs more CPU cycles to be
processed than the standard sample. Similarly, for a video frame with a lower
resolution of $s_{n}^{2}$ pixels and $s_{n}<s_{\text{standard}}$, it will need
fewer CPU cycles than the standard sample.

To perform object detection, our metaverse system incorporates You Look Only
Once (YOLO) algorithm [36], whose main structure consists of convolutional
neural networks (CNNs) [37]. From [38], the time complexity of a CNN can be
given as

$\displaystyle\mathcal{O}(\sum_{l=1}^{L}c_{l-1}\cdot k^{2}\cdot c_{l}\cdot
m_{l}^{2}),$ (5)

where $L$ is the total number of convolutional layers; $k$ refers to the
kernel length; for $l\in\\{1,2,\ldots,L\\}$, $c_{l-1}$ is the number of input
channels to the $l$-th layer, $c_{l}$ is the number of kernels (i.e., the
number of output channels after the $l$-th layer), and $m_{l}$ means the
output feature map’s spatial size. From [38], in the first layer, $m_{1}$ can
be computed by

$\displaystyle m_{1}=(s_{n}-k+2*\texttt{padding})/\texttt{stride}+1.$ (6)

In the above equation, padding and stride are set in a CNN [38], and
$s_{n}\times s_{n}$ is the resolution of the input image on MAR device $n$. In
CNN, the current layer’s output feature map is the next layer’s input. Thus,
the input image size affects the spatial size of output feature maps of
subsequent layers. Fig. 2 is a simple illustration of the first convolutional
layer in a CNN to explain how we get the time complexity in (5).

Figure 2: Illustration of the 1st convolutional layer in a CNN. In this
example, padding is 0 and stride is 1. $c_{0}$ is the default number of
channels (e.g., in a color image, we have 3 channels). $c_{1}$ is the number
of kernels, the same as the number of output channels. $m_{1}$ is calculated
by Eq. (6). By applying the kernel to the input image, doing element-wise
multiplication and adding each element weighted by the kernel, we get the
output by $c_{0}k^{2}m_{1}^{2}$ calculations. Since we have $c_{1}$ kernels,
the total number of calculations is $c_{0}k^{2}c_{1}m_{1}^{2}$. If we have $L$
convolutional layers, we can get the time complexity in (5).

Because we want to find the relationship between the CPU cycles and input
images, we only focus on how the input images affect the time complexity.
Therefore, from Eq. (5) and (6), we find the time complexity with respect to
the input image resolution $s_{n}$ is $\mathcal{O}(s_{n}^{2})$. Recall that
$c_{n}$ is the number of CPU cycles on device $n$ per standard sample with a
resolution of $s_{\text{standard}}\times s_{\text{standard}}$ pixels. Hence,
for simplicity, we just consider that the number of CPU cycles on device $n$
to process a video frame with a resolution of $s_{n}\times s_{n}$ pixels is
given by $\zeta s_{n}^{2}\times c_{n}$ for
$\zeta:=\frac{1}{s_{\text{standard}}^{2}}$. Then the required CPU cycles to
process $D_{n}$ video frames on device $n$ is

$\displaystyle\zeta s_{n}^{2}c_{n}D_{n},$ (7)

so the local computation energy consumption in $R_{l}$ local iterations for
one global round is

$\displaystyle E_{n}^{cmp}=\kappa R_{l}\zeta s_{n}^{2}c_{n}D_{n}f_{n}^{2},$
(8)

where $R_{l}$ is the number of local iterations between two global
aggregations. Let $R_{g}$ be the number of global rounds. The total energy
consumption is

$\displaystyle\mathcal{E}=R_{g}\sum_{n=1}^{N}(E_{n}^{trans}+E_{n}^{cmp}).$ (9)

###  III-B Total Completion Time

The total completion time, $\mathcal{T}$, of the FL system model over the
wireless network comprises local computation time and data transmission time.
The local computation time of device $n$ in one global iteration,
$T^{cmp}_{n}$, is

$\displaystyle\textstyle{T_{n}^{cmp}=R_{l}\frac{\zeta
s_{n}^{2}c_{n}D_{n}}{f_{n}}},$ (10)

where $R_{l}$ is the number of local iterations on each device $n$, $\zeta
s_{n}^{2}c_{n}D_{n}$ is the number of CPU cycles, and $f_{n}$ refers to the
CPU frequency (i.e., the number of CPU cycles per second).

Then, with the transmission time given by (2), the total completion time
becomes

$\displaystyle\textstyle{\mathcal{T}=R_{g}\max_{n\in\mathcal{N}}\\{T_{n}^{cmp}+T_{n}^{up}\\}}.$
(11)

As we have mentioned, since the base station’s output power is larger than the
maximum uplink transmission power of a user device, the downlink time is much
smaller than the uplink time. Hence, we ignore the downlink time, as in other
federated learning work [8, 7, 9].

###  III-C Accuracy Model

An accuracy metric $\mathcal{A}$ is considered in the optimization objective.
Since the optimization variables include the video frame resolution $s_{n}$
and if our goal is to just minimize the weighted sum of total energy
consumption $\mathcal{E}$ and total delay $\mathcal{T}$, less $s_{n}$ is
always better. Hence, we incorporate $\mathcal{A}$ into the objective function
to make the optimization of $s_{n}$ less trivial.

Let $\mathcal{A}$ be the final training accuracy of the whole federated
learning process and consider it as a function of $s_{1},s_{2},\ldots,s_{N}$.
This can be written as $\mathcal{A}(s_{1},s_{2},\ldots,s_{N})$ for simplicity.

We use the result (12) below from [11] to represent
$\mathcal{A}(s_{1},s_{2},\ldots,s_{N})$. In [11], Liu et al. formulated the
model accuracy as a function of the video frame resolution. They conducted
experiments based on YOLO algorithm [36] to construct the accuracy function
concerning different video frame resolutions. The accuracy function introduced
in [11] and adopted by a later work [39] is

$\displaystyle 1-1.578e^{-6.5\times 10^{-3}s_{n}},$ (12)

whose concavity is exploited in [11, 39]. In our solution to the optimization
problem later, we do not need any requirement on the convexity/concavity of
$A_{n}|_{n\in\mathcal{N}}$, since we will linearize the expression of
$A_{n}|_{n\in\mathcal{N}}$ in Section V-A later.

##  IV Optimization Problem Formulation

This section formulates the optimization problem. We obtain the formulation by
discussing the optimization objective function, optimization variables, and
constraints, respectively.

###  IV-A Optimization Problem

For our federated learning with augmented reality, the optimization problem is
as follows (the meanings of the symbols will be explained below):

$\displaystyle\min_{\\{B_{n},p_{n},f_{n},s_{n}\\}_{n\in\mathcal{N}}}$
$\displaystyle~{}w_{1}\mathcal{E}+w_{2}\mathcal{T}-\rho\mathcal{A},$ (13)
subject to: $\displaystyle\sum_{n=1}^{N}B_{n}\leq B,~{}B_{n}\geq 0,~{}\forall
n\in\mathcal{N},$ (13a) $\displaystyle p_{n}^{min}\leq p_{n}\leq
p_{n}^{max},~{}\forall n\in\mathcal{N},$ (13b) $\displaystyle f_{n}^{min}\leq
f_{n}\leq f_{n}^{max},~{}\forall n\in\mathcal{N},$ (13c) $\displaystyle
s_{n}\in\\{s_{min},s_{max}\\},~{}\forall n\in\mathcal{N},$ (13d)

where constraint (13a) defines the upper limit of total bandwidth, (13b) sets
the range of each device’s transmission power, (13c) is the range of CPU
frequency of each device, and (13d) gives two choices for the video frame
resolution (future work can consider more values). Note that $s_{n}^{min}$ and
$s_{n}^{max}$ are constants. In the objective function (13), the total energy
consumption $\mathcal{E}$ is given in Eq. (9), while the total delay
$\mathcal{T}$ is given in Eq. (11) and the accuracy function is defined in Eq.
(12). The weight parameters $w_{1}$, $w_{2}$, and $\rho$ are all non-negative,
and we enforce $w_{1}+w_{2}=1$ without loss of generality. For devices in a
low-battery condition, we can have $w_{1}=1$ and $w_{2}=0$. In contrast, if
the system is delay-sensitive, we can choose $w_{1}=0$ and $w_{2}=1$. $\rho$
is the weight parameter of the model accuracy. Larger (resp., smaller) $\rho$
means that the system puts more (resp., less) emphasis on the accuracy part.

Reasons for enforcing the accuracy metric $\mathcal{A}$ as
$\sum_{n=1}^{N}A_{n}(s_{n})$ and constraining $s_{n}$ via (13d). The
optimization for other variables $B_{n},p_{n},f_{n}$ is already quite complex,
as shown in our recent conference version [1]. A complex expression for
$\mathcal{A}$ or more value choices for $s_{n}$ can cause the overall
optimization to be intractable. Besides, our simple consideration for
$\mathcal{A}$ and $s_{n}$ is the first step. Future directions include
considering more complex $\mathcal{A}$ and more value choices $s_{n}$. We
would like to mention that “$-\rho\sum_{n=1}^{N}A_{n}(s_{n})$” as a part of
the optimization objective function has also been considered in recent work
[11, 39] on augmented reality, but [11, 39] are not about federated learning
and have simpler optimization problems compared with ours.

Before solving the optimization problem, we now transform the problem into a
form that is easier to handle. From Eq. (11), $\mathcal{T}$ equals
$R_{g}\max_{n\in\mathcal{N}}\\{T_{n}^{cmp}+T_{n}^{up}\\}$. Such max function
in the above expression of $\mathcal{T}$ makes the minimization in (13) hard
to solve. To circumvent this difficulty, we introduce an auxiliary variable
$T$, replace $\mathcal{T}$ by $R_{g}T$, and add an extra constraint
$T_{n}^{cmp}+T_{n}^{up}\leq T$ for $n\in\mathcal{N}$. Then, the problem writes
as follows:

$\displaystyle\min_{B_{n},p_{n},f_{n},s_{n},T}$
$\displaystyle~{}w_{1}R_{g}\sum_{n=1}^{N}(p_{n}T_{n}^{up}+\kappa R_{l}\zeta
s_{n}^{2}c_{n}D_{n}f_{n}^{2})+w_{2}R_{g}T-\rho\sum_{n=1}^{N}A_{n}(s_{n})$ (14)
subject to: (13a), (13b), (13c), (13d), $\displaystyle
T_{n}^{cmp}+T_{n}^{up}\leq T,~{}\forall n\in\mathcal{N}.$ (14a)

Difficulty of solving problem (14). Since the formulation of $T_{n}^{up}$ and
some constraints are not convex with respect to the optimization variables,
problem (14) is non-convex, which is typically difficult to solve.
Consequently, we must explore other ways to transform this problem to make it
solvable.

###  IV-B Problem Decomposition

To continuously transform the problem (14) into a simpler form, we thereby
separate problem (14) into two subproblems. One has $p_{n}$ and $B_{n}$ as
optimization variables since $p_{n}$ and $B_{n}$ are in the same term, and the
other has $f_{n}$, $s_{n}$ and $T$ as optimization variables.

Two minimization problems are as follows:

$\displaystyle\text{{Subproblem 1 (SP1):}}~{}\min_{f_{n},s_{n},T}$
$\displaystyle~{}w_{1}R_{g}\sum_{n=1}^{N}\kappa R_{l}\zeta
s_{n}^{2}c_{n}D_{n}f_{n}^{2}+w_{2}R_{g}T-\rho\sum_{n=1}^{N}A_{n}(s_{n})$ (15)
subject to: (13c),(13d),(14a). $\displaystyle\text{{Subproblem 2
(SP2):}}~{}\min_{p_{n},B_{n}}$
$\displaystyle~{}w_{1}R_{g}\sum_{n=1}^{N}p_{n}\frac{d_{n}}{B_{n}\log_{2}(1+\frac{p_{n}g_{n}}{N_{0}B_{n}})}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
(16) subject to: (13b), (13a),(14a).

In the rest of the paper, for a vector $\bm{v}$, its $n$-th dimension is
denoted as $v_{n}$. Based on the above notation system, we have
$\bm{p}:=[p_{1},p_{2},\ldots,p_{N}]$, $\bm{B}:=[B_{1},B_{2},\ldots,B_{N}]$,
$\bm{f}:=[f_{1},f_{2},\ldots,f_{N}]$, and
$\bm{s}:=[s_{1},s_{2},\ldots,s_{N}]$. Our algorithm to solve the problem (14)
is based on the Block coordinate descent (BCD) technique. BCD optimizes a
group of optimization variables (or blocks of indices) and fixes other blocks
at each iteration [40]. We randomly choose feasible $(\bm{p},\bm{B})$ at
first, and solve Subproblem 1 to get the optimal ($\bm{f}$, $\bm{s}$) (note
that $T$ is an auxiliary variable). Then, we solve Subproblem 2 to get the
optimal $(\bm{p},\bm{B})$. The whole process consists of iteratively solving
Subproblems 1 and 2 until convergence.

##  V Solution of The joint optimization problem

In this section, we will explain how to solve Subproblems 1 and 2.

###  V-A Solution to Subproblem 1

Since the video frame resolution $s_{n}$ is a discrete variable, we relax it
into a continuous variable $\hat{s_{n}}$ to make Subproblem 1 easier to solve.
Next, we handle $A_{n}(\hat{s_{n}})$. As the first step to studying FL-MAR, we
propose a simple yet powerful linearization approach which does not consider
any property of the function $A_{n}$. In the future, we can consider more
specific properties, such as the convexity/concavity of $A_{n}$, to design
better algorithms. By using two points $(s_{\min},A_{n}(s_{\min}))$ and
$(s_{\max},A_{n}(s_{\max}))$, we approximate $A_{n}(\hat{s_{n}})$ as the
following linear function $\hat{A}_{n}(\hat{s_{n}})$:

$\displaystyle\hat{A}_{n}(\hat{s_{n}}):=\hat{k}_{n}(\hat{s_{n}}-s_{\min})+A_{n}(s_{\min}),$
(17)

where
$\hat{k}_{n}:=\frac{A_{n}(s_{\max})-A_{n}(s_{\min})}{s_{\max}-s_{\min}}$.

Then, Subproblem 1 becomes

SP1_v1: $\displaystyle\min_{f_{n},\hat{s_{n}},T}w_{1}R_{g}\sum_{n=1}^{N}\kappa
R_{l}\zeta\hat{s_{n}}^{2}c_{n}D_{n}f_{n}^{2}+w_{2}R_{g}T-\rho\sum_{n=1}^{N}\hat{A}_{n}(\hat{s_{n}})$
(18) subject to:
$\displaystyle~{}\text{(\ref{fminfmax}),}~{}\text{(\ref{constra:time}),}~{}s_{\min}\leq\hat{s_{n}}\leq
s_{\max},~{}\forall n\in\mathcal{N}.$ (18a)

It is easy to verify that the objective function and constraints of variables
$f_{n}$, $\hat{s_{n}}$ and $T$ in (18) are convex. Then, we treat the
optimization problem by the Karush–Kuhn–Tucker (KKT) approach.

Since applying KKT conditions is a common approach in convex optimization, we
place the derivation in Appendix B. In short, we first introduce the Lagrange
function of SP1_v1 and define the Lagrange multiplier
$\bm{\lambda}=[\lambda_{1},\cdots,\lambda_{N}]$. After applying KKT conditions
to the Lagrange function, we derive the expression of $\bm{f}$ in
$\bm{\lambda}$ and thus derive the dual problem of SP_v1. In this paper, we
use CVX [41] to solve the dual problem and get the optimal
$\bm{\lambda^{*}}=[\lambda_{1}^{*},...,\lambda_{n}^{*}]$. Then, we are able to
calculate $\bm{f}$ and $\bm{\hat{s}}$ through Eq. (A.5) (in Appendix B).

Due to $f_{n}^{\min}\leq f_{n}\leq f_{n}^{\max}$, $f_{n}$ is set as follows
with $f_{n}^{*}$ in Eq. (A.5) of [42]:

$\displaystyle f_{n}$
$\displaystyle=\min(f_{n}^{\max},\max(f_{n}^{*},~{}f_{n}^{\min})).$ (19)

Since the video frame resolution $s_{n}$ is discrete, we map $\hat{s_{n}}$ to
$s_{n}$ according to the formula below

$\displaystyle
s_{n}=\begin{cases}&s_{\max},~{}\text{if}~{}\hat{s_{n}}\geq\frac{s_{\min}+s_{\max}}{2},\\\\[-10.0pt]
&s_{\min},~{}\text{if}~{}\hat{s_{n}}<\frac{s_{\min}+s_{\max}}{2},\end{cases}\text{where
$\hat{s_{n}}$ is in Eq. (\ref{relation_f_s_lamda}) of Appendix B.}$ (20)

There may exist other methods to map $\hat{s_{n}}$ to $s_{n}$, which may be
more involved and require more computations. Our proposed mapping in Eq. (20)
above is simple and does not require complex computations. The effectiveness
of our mapping is further confirmed by experimental results in Section VII.
When $\hat{s_{n}}$ equals $\frac{s_{\min}+s_{\max}}{2}$, $s_{n}$ can be either
$s_{\max}$ or $s_{\min}$, while this paper just selects $s_{\max}$ as in Eq.
(20) for simplicity.

###  V-B Subproblem 2

Subproblem 2 is a typical sum-of-ratios minimization problem, which is NP-
complete, and thus very hard to solve [14]. Firstly, we define
$G_{n}(p_{n},B_{n}):=B_{n}\log_{2}(1+\frac{p_{n}g_{n}}{N_{0}B_{n}})$ to
simplify the notations. Additionally, (14a) can be written as $r_{n}\geq
r_{n}^{\min}$, where
$r_{n}^{\min}=\frac{d_{n}}{T-\frac{R_{l}c_{n}D_{n}}{f_{n}}}$. In fact,
$G_{n}(p_{n},B_{n})$ is the same as the data rate $r_{n}$ defined in (1).

Then, we transform Subproblem 2 into the epigraph form through adding an
auxiliary variable $\beta_{n}$ and let
$\frac{p_{n}d_{n}}{G_{n}(p_{n},B_{n})}\leq\beta_{n}$. Then, we get an
equivalent problem called SP2_v1.

SP2_v1:
$\displaystyle~{}\min_{p_{n},B_{n},\beta_{n}}w_{1}R_{g}\sum_{n=1}^{N}\beta_{n}$
(21) subject to
$\displaystyle\text{(\ref{constra:bandwidth})},~{}\text{(\ref{pminpmax})},~{}$
$\displaystyle G_{n}(p_{n},B_{n})\geq r_{n}^{\min},~{}\forall
n\in\mathcal{N},$ (21a) $\displaystyle
p_{n}d_{n}-\beta_{n}G_{n}(p_{n},B_{n})\leq 0,~{}\forall n\in\mathcal{N},$
(21b)

where $\beta_{n}$ is an auxiliary variable. Because
$\frac{p_{n}d_{n}}{G_{n}(p_{n},B_{n})}\leq\beta_{n}$ and
$G_{n}(p_{n},B_{n})>0$, we can get constraint (21b). However, due to
constraint (21b), problem SP2_v1 is non-convex, which is still intractable. To
continue transforming this problem, we need to give the following lemma.

######  Lemma 1 (Proved in Appendix C).

$G_{n}(p_{n},B_{n})$ is a concave function.

The original objective function in Subproblem 2 is the sum of fractional
functions. It is obvious that $p_{n}d_{n}$ is convex, and Lemma 1 reveals
$G_{n}(p_{n},B_{n})$ is concave. With such a trait, problem SP2_v1 can be
transformed into a subtractive-form problem via the following theorem.

######  Theorem 1.

If $(\bm{p}^{*},\bm{B}^{*},\bm{\beta}^{*})$ is the solution of problem SP2_v1,
there exists $\bm{\nu}^{*}$ satisfying that $(\bm{p}^{*},\bm{B}^{*})$ is a
solution of the following problem with $\bm{\nu}=\bm{\nu}^{*}$ and
$\bm{\beta}=\bm{\beta}^{*}$.

SP2_v2:
$\displaystyle~{}\textstyle{\min_{p_{n},B_{n}}\sum_{n=1}^{N}\nu_{n}(p_{n}d_{n}-\beta_{n}G_{n}(p_{n},B_{n}))}$
(22) subject to:
$\displaystyle~{}\text{(\ref{constra:bandwidth})},~{}\text{(\ref{pminpmax})},~{}\text{(\ref{constra:subproblem2_data_rate_min}).}$

Moreover, under the condition of $\bm{\nu}=\bm{\nu}^{*}$ and
$\bm{\beta}=\bm{\beta}^{*}$, ($\bm{p}^{*},\bm{B}^{*}$) satisfies the following
equations.

$\displaystyle\textstyle{\nu_{n}^{*}=\frac{w_{1}R_{g}}{G_{n}(p_{n}^{*},B_{n}^{*})},~{}\text{for}~{}n=1,\cdots,N,\text{
and }\beta_{n}^{*}=\frac{p_{n}^{*}d_{n}}{G_{n}(p_{n}^{*},B_{n}^{*})},\text{
for}~{}n=1,\cdots,N.}$ (23)

Proof: The proof follows directly from Lemma 2.1 of [14].

###  V-C Solution to Subproblem 2

Theorem 1 indicates that problem SP2_v2 has the same optimal solution
$(\bm{p^{*},\bm{B}^{*}})$ with problem SP2_v1. Consequently, we can first
provide $(\bm{\nu},\bm{\beta})$ to obtain a solution $(p_{n},B_{n})$ through
solving problem SP2_v2. Then, the second step is to calculate
$(\bm{\nu},\bm{\beta})$ according to (23). The first step can be seen as an
inner loop to solve problem SP2_v2 and the second step is the outer loop to
obtain the optimal $(\bm{\nu},\bm{\beta})$. The iterative optimization is
given in Algorithm 1, which is a Newton-like method. The original algorithm is
provided in [14]. In Algorithm 1, we define

$\displaystyle\phi_{1}(\bm{\beta})\\!=\\![-p_{n}d_{n}+\beta_{n}G_{n}(p_{n},B_{n})|_{n=1,\cdots,N}]^{T}\\!,~{}\phi_{2}(\bm{\nu})\\!=\\![-w_{1}R_{g}+\nu_{n}G_{n}(p_{n},B_{n})|_{n=1,\cdots,N}]^{T},$
(24)

which are two vectors in $\mathbb{R}^{N}$, and
$\phi(\bm{\beta},\bm{\nu})=[\phi_{1}^{T}(\bm{\beta}),\phi_{2}^{T}(\bm{\nu})]^{T}$,
which is a vector in $\mathbb{R}^{2N}$. Additionally, the Jacobian matrices of
$\phi_{1}(\bm{\beta})$ and $\phi_{2}(\bm{\nu})$ are

$\displaystyle\phi_{1}^{\prime}(\bm{\beta})=diag(G_{n}(p_{n},B_{n})|_{n=1,\cdots,N}),~{}\phi_{2}^{\prime}(\bm{\nu})=diag(G_{n}(p_{n},B_{n})|_{n=1,\cdots,N}),$
(25)

where $diag()$ represents a diagonal matrix. [14] proves that when
$\phi(\bm{\beta},\bm{\nu})=\bm{0}$, the optimal solution
$(\bm{\nu},\bm{\beta})$ is achieved. Simultaneously, it satisfies (23). Thus,
a Newton-like method can be used to update $(\bm{\nu},\bm{\beta})$, which is
done in (28).

1 Initialize $i=0$, $\xi\in(0,1)$, $\epsilon\in(0,1)$. Given feasible
($\bm{p}^{(0)}$, $\bm{B}^{(0)}$).

2 repeat

3 Calculate
$\nu_{n}^{(i)}=\frac{w_{1}R_{g}}{G_{n}(p_{n}^{(i)},B_{n}^{(i)})},~{}\beta_{n}^{(i)}=\frac{p_{n}d_{n}}{G_{n}(p_{n}^{(i)},B_{n}^{(i)})}$
for $n=1,\cdots,N$.

4 Obtain $(\bm{p}^{(i+1)}$, $\bm{B}^{(i+1)})$ through solving SP2_v2 according
to Theorem 2 by using CVX given $(\bm{\nu}^{(i)},\bm{\beta}^{(i)})$.

5 Let $j$ be the smallest integer which satisfies
$\displaystyle|\phi(\bm{\beta}^{(i)}+\xi^{j}\bm{\sigma_{1}}^{(i)},\bm{\nu}^{(i)}+\xi^{j}\bm{\sigma_{2}}^{(i)})|\leq(1-\epsilon\xi^{j})|\phi(\bm{\beta}^{(i)},\bm{\nu}^{(i)})|,$
(26) where
$\displaystyle\bm{\sigma_{1}}^{(i)}=-[\phi_{1}^{\prime}(\bm{\beta}^{(i)})]^{-1}\phi_{1}(\bm{\beta}^{(i)}),$
$\displaystyle\bm{\sigma_{2}}^{(i)}=-[\phi_{2}^{\prime}(\bm{\nu}^{(i)})]^{-1}\phi_{2}(\bm{\nu}^{(i)}).$
(27)

6 Update
$\displaystyle(\bm{\beta}^{(i+1)},\bm{\nu}^{(i+1)})=(\bm{\beta}^{(i)}+\xi^{j}\bm{\sigma_{1}}^{(i)},\bm{\nu}^{(i)}+\xi^{j}\bm{\sigma_{2}}^{(i)}).$
(28)

7 Let $i\leftarrow i+1$.

8until  _$\phi(\bm{\beta}^{(i)},\bm{\nu}^{(i}))=\bm{0}$ or the number of
iterations reaches the maximum iterations $i_{0}$_;

Algorithm 1 Optimization of $(\bm{p},\bm{B})$ and $(\bm{\nu},\bm{\beta})$

Until now, the remaining problem needed to focus on is problem SP2_v2.
Remember $\bm{\nu}$ and $\bm{\beta}$ are fixed in this problem, which are seen
as constants. Given Lemma 1, $p_{n}d_{n}$ is convex, and three constraints
(13b) (13a) (21a) are convex, the objective function (22) in problem SP2_v2 is
convex. Thus, KKT conditions are sufficient and necessary optimality
conditions for finding the optimal solution.

Since the process of applying KKT conditions to SP2_v2 is similar with that of
SP1_v1 in (18), we skip the derivation here and put it in Appendix D.

Then, Theorem 2 is given to find the optimal bandwidth and transmission power
for each device. Note that $\tau_{n}|_{n=1,\ldots,N}$ and $\mu$ are non-
negative Lagrange multipliers and the Lagrange function of SP2_v2 is written
in Appendix D.

######  Theorem 2.

The optimal bandwidth $\bm{B}$ and transmission power $\bm{p}$ are expressed
as

$\displaystyle
B_{n}^{*}=\begin{cases}\frac{r_{n}^{\min}}{\log_{2}(1+\Lambda_{n})},~{}\text{if}~{}\tau_{n}\neq
0,\\\\[-5.0pt] \text{Solution~{}to~{}problem~{}(\ref{SP2_v3}) in Appendix D
of~{}\cite[cite]{[\@@bibref{}{full}{}{}]},}~{}\text{if}~{}\tau_{n}=0,\end{cases}$
$\displaystyle\text{and
}p_{n}^{*}=\min(p_{n}^{\max},\max(\Gamma(B_{n}),~{}p_{n}^{\min})),~{}\text{when}~{}\mu\neq
0,~{}n=1,\cdots,N,$

where
$\Lambda_{n}=\frac{(\nu_{n}\beta_{n}+\tau_{n})g_{n}}{N_{0}d_{n}\nu_{n}\ln 2}$
and
$\Gamma(B_{n})=(\frac{(\nu_{n}\beta_{n}+\tau_{n})g_{n}}{N_{0}d_{n}\nu_{n}\ln
2}-1)\frac{N_{0}B_{n}}{g_{n}}.$

We prove Theorem 2 in Appendix D. Then problem SP2_v2 is solved.

###  V-D Resource Allocation Algorithm

Here we give the complete resource allocation algorithm based on BCD in
Algorithm 2.

1 Initialize $sol^{(0)}=(\bm{p}^{(0)},\bm{B}^{(0)},\bm{f}^{(0)},\bm{s}^{(0)})$
of problem (14), iteration number $k=1$.

2 repeat

3 Solve Subproblem 1 through solving problem (A.7) with CVX given
$(\bm{p}^{(k-1)},\bm{B}^{(k-1)})$. Obtain $(\bm{f}^{(k)},\bm{s}^{(k)})$
according to Eq. (19) and Eq. (20).

4 Solve Subproblem 2 through Algorithm 1 and obtain
$(\bm{p}^{(k)},\bm{B}^{(k)})$ with CVX.

5 $sol^{(k)}=(\bm{p}^{(k)},\bm{B}^{(k)},\bm{f}^{(k)},\bm{s}^{(k)})$.

6 Set $k\leftarrow k+1$.

7until  _$|sol^{(k)}-sol^{(k-1)}|\leq\epsilon_{0}$ or the number of iterations
achieves $K$_;

Algorithm 2 Resource Allocation Algorithm of FL-MAR system for the Metaverse.

Algorithm 2 first initializes a feasible solution within the range of power
$\bm{p}$ and bandwidth $\bm{B}$, and the sum of the bandwidth of each device
cannot exceed $B$. Then, solving problem (A.7) with given $(\bm{p},\bm{B})$
makes sure we get optimal $\bm{f}$ at this step. Subproblem 2 does not include
$\bm{f}$ and it is an optimization problem focusing on $\bm{p}$ and $\bm{B}$.

##  VI Convergence and Time Complexity

In this section, we analyze the convergence of our resource allocation
algorithm (Algorithm 2) and present the time complexity of Algorithm 2.

###  VI-A Convergence Analysis

Note that Algorithm 2 iteratively solves Subproblem 1 to get ($\bm{f},\bm{s}$)
and Subproblem 2 to get ($\bm{p},\bm{B}$). Subproblem 1 will get
($\bm{f},\bm{s}$) through solving its dual problem. Then, we should analyze
the convergence of Algorithm 1, which is to solve Subproblem 2.

In [1], we show that Algorithm 1 converges with a linear rate at any starting
point in a solution set and a quadratic convergence rate of the solution’s
neighborhood. Therefore, iteratively solving Subproblem 1 to get
($\bm{f},\bm{s}$) and Subproblem 2 to get ($\bm{p},\bm{B}$) will converge.

###  VI-B Time Complexity

For Algorithm 1, we use floating point operations (flops) to count the time
complexity. One real addition/multiplication/division is counted as one flop
[43]. We mainly focus on analyzing Steps 3–6 because they take a major part of
computational complexity. Step 3 needs $2N$ flops. Step 4 utilizes CVX to
solve problem SP2_v2.Besides, CVX invokes an interior-point algorithm to solve
Semidefinite Programming (SDP) problems. Thus, given the solution accuracy
$\epsilon_{1}>0$, the worst time complexity is
$\mathcal{O}(N^{4.5}\log\frac{1}{\epsilon_{1}})$ [44, 45]. Additionally, Step
5 will take $\mathcal{O}((j+1)N)$ flops, where $j$ is the smallest integer
satisfying (26). Our experiments show that typically $j$ is a small number. If
$j=0$ (i.e., $\xi^{j}=1$), Step 6 is just the Newton method. Otherwise, it is
a Newton-like method. Obviously, Step 6 takes $4N$ flops. Since the number of
iterations is not larger than $i_{0}$, the time complexity of Algorithm 1 will
not exceed $\mathcal{O}(i_{0}(N^{4.5}\log\frac{1}{\epsilon_{1}}+(j+1)N))$.

Algorithm 2 solves two subproblems successively by using the convex package
CVX in each iteration. Steps 3 and 4 both use CVX to solve two subproblems. As
we have mentioned above, the time complexity caused by CVX is
$\mathcal{O}(N^{4.5}\log\frac{1}{\epsilon_{1}})$ given a solution accuracy
$\epsilon_{1}>0$. Step 4 invokes Algorithm 1, and we have already analyzed the
time complexity of Algorithm 1. Finally, the overall complexity of Algorithm 2
is $\mathcal{O}(K(i_{0}+1)N^{4.5}\log\frac{1}{\epsilon_{1}}+Ki_{0}(j+1)N)$.

##  VII Experiments

We conduct extensive experiments to illustrate the effectiveness of our
proposed algorithm. We first present the parameter setting of our experiments
in Section VII-A, and then report various experimental results (including
comparison with other work) in other subsections.

###  VII-A Parameter Setting

We describe the parameter settings of the experiments as follows. $N$ denoting
the number of devices is 50. The devices are uniformly located in a circular
area of size $500$ m $\times$ $500$ m, and the center is a base station.
Between each device and the base station, the wireless channel’s pass loss is
modeled as $128.1+37.6\log_{10}(\texttt{distance})$ along with the standard
deviation of shadow fading, which is $8$ dB, where distance denotes the
distance from the device to the base station with kilometer as the unit. The
power spectral density of Gaussian noise $N_{0}$ is $-174$ dBm/Hz.

Additionally, the number of local iterations $R_{l}$ is set as $10$ and the
number of global aggregations $R_{g}$ is 100. The data size $d_{n}$ for each
device to upload is set as $28.1$ kbits, based on the result of [7]. Besides,
each device has $D_{n}=500$ samples. The number of CPU cycles per sample,
$c_{n}$, is chosen randomly from $[1,3]\times 10^{4}$. The effective switched
capacitance $\kappa$ is $10^{-28}$. The maximum CPU frequency $f_{n}^{\max}$
and the maximum transmission power $p_{n}^{\max}$ for each device $n$ are $2$
GHz and $12$ dBm, respectively, so we just write them as $f^{\max}$ and
$p^{\max}$ below. The minimum CPU frequency $f_{n}^{\min}$ is $0$ Hz, and the
minimum transmission power $p_{n}^{\min}$ is $0$ dBm. The total bandwidth $B$
is $20$ MHz. By default, $s_{\min}=160$ pixels, $s_{\max}=640$ pixels and
$s_{\text{standard}}=160$ pixels. Recall that the constant $\zeta$ equals
$\frac{1}{s_{\text{standard}}^{2}}$.

In all experiments, we “normalize”111The intuition for doing this is that
since $\mathcal{E}$ and $\mathcal{T}$ can be understood as “costs” while
$\mathcal{A}$ can be understood as “gain”, then with $w_{1}+w_{2}=1$, the
combined coefficient $w_{1}+w_{2}$ for the cost part is $1$, for better
clarity. the weight parameters such that $w_{1}+w_{2}=1$; i.e., if
$w_{1}+w_{2}$ does not equal to $1$ at the beginning, we just divide all the
parameters by $w_{1}+w_{2}$. Note that the above normalization can apply to
all possible $(w_{1},w_{2},\rho)$ except for $(w_{1}=0,w_{2}=0,\rho=1)$ since
it corresponds to the trivial problem of just maximizing the accuracy part
$\mathcal{A}$.

###  VII-B How does Our Optimization Algorithm Perform with respect to System
Parameters?

In our optimization, we have three weight parameters $w_{1}$, $w_{2}$, and
$\rho$, where $w_{1}+w_{2}=1$ as explained above. If $w_{1}$ is greater than
$w_{2}$, energy consumption will take a more dominant position in our
optimization problem. If $w_{2}$ is larger than $w_{1}$, it is more important
to optimize the delay instead of energy. Analogously, if the weight parameter
$\rho$ is larger, the object detection accuracy will take a more dominant
position. To explore the effect of weight parameters of total energy and time
consumption and model accuracy separately, we first show experiments under
different $(w_{1},w_{2})$ with fixed $\rho$ and then show results under
different $\rho$ with fixed $(w_{1},w_{2})=(0.5,0.5)$. The algorithms to be
compared with are summarized in Table I (comparison with other work is in
Section VII-D).

TABLE I: Algorithm Comparison | resource allocation ($f,p,B$) | video frame resolution $s_{n}$
---|---|---
Proposed algorithm | optimized | optimized
Benchmark (MinPixel) | random | fixed
RandPixel | random | random

####  VII-B1 Different ($w_{1}$, $w_{2}$) with fixed $\rho$

In the experiments, we compare three pairs of $(w_{1},w_{2})=(0.9,0.1)$,
$(0.5,0.5)$, and $(0.1,0.9)$ with a benchmark algorithm under two scenarios.
One is under different maximum transmission powers, and the other is under
different maximum computational frequencies. $(w_{1},w_{2})=(0.9,0.1)$ is to
simulate devices under a low-battery condition, so the weight parameter of the
energy consumption is set as 0.9. $(w_{1},w_{2})=(0.5,0.5)$ represents the
normal scenario, which means no preference to the time or energy. Besides,
$(w_{1},w_{2})=(0.1,0.9)$ stands for the time-sensitive scenario. Thus, the
algorithms in the comparison include:

  * •

Under weights $(w_{1}=0.9,w_{2}=0.1)$, $(w_{1}=0.5,w_{2}=0.5)$, or
$(w_{1}=0.1,w_{2}=0.9)$.

  * •

Benchmark algorithm (MinPixel): 1) When comparing energy consumption and
completion time under different maximum transmission powers, for the $n$-th
device, randomly select the CPU frequency $f_{n}$ from 0.1 to 2 GHz and set
$p_{n}=p^{\max}$, $B_{n}=\frac{B}{N}$ and $s_{n}=s_{\min}$. 2) When comparing
under different maximum CPU frequencies, randomly select the transmission
power $p_{n}$ between 0 and 12 dBm and set $f_{n}=f^{\max}$,
$B_{n}=\frac{B}{N}$ and $s_{n}=s_{\min}$.

When comparing energy consumption and completion time under different maximum
transmission powers, $\rho=1$. When comparing under different maximum CPU
frequencies, $\rho=10$. To be more general, we run our algorithm at each pair
of weight parameters 100 times and take the average value of energy and time
consumption. At each time, 50 users are generated randomly.

Fig. 3(a) and Fig. 3(b) show the results of total energy consumption and total
time consumption with three pairs of $(w_{1},w_{2})$ under different maximum
transmit power limits, respectively. It can be observed when $w_{1}$ becomes
larger and $w_{2}$ decreases, the total energy consumption becomes smaller,
and the total time becomes larger. The reason is that if $w_{1}$ (resp.,
$w_{2}$) increases, our optimization algorithm will emphasize minimizing the
energy cost (resp., time consumption).

Additionally, in Fig. 3(a), the green, blue and red lines are all below the
benchmark line with a wide gap. Note that the benchmark algorithm sets all
video frame resolutions as $s_{\min}$, which means it saves much computational
energy cost while sacrificing object detection accuracy. Even in this
situation, our algorithm obtains better total energy consumption than the
benchmark even at $w_{1}=0.1$ (this means the algorithm mainly focuses on
minimizing the total time consumption).

(a) Total energy consumption with different weight parameters under different
maximum transmit power limits. $\rho=1$.

(b) Total time with different weight parameters under different maximum
transmit power limits. $\rho=1$.

(c) Total energy consumption with different weight parameters under different
maximum CPU frequencies. $\rho=10$.

(d) Total time with different weight parameters under different maximum CPU
frequencies. $\rho=10$.

Figure 3: Experiments varying maximum transmit power limits / computational
frequencies.

Correspondingly, Fig. 3(b) reveals that our time optimization is still better
than the benchmark except for $w_{1}=0.9$ and $w_{2}=0.1$ after $p^{\max}=8$
dBm. For $w_{1}=0.9$ and $w_{2}=0.1$ (the red line) in Fig. 3(b), it stresses
the importance of energy consumption, so it tries to find a smaller weighted
sum of energy and time consumption especially for the energy consumption.
Thus, after $p^{\max}=8$ dBm, its time cost is slightly higher than the
benchmark, but the energy consumption is much smaller than the benchmark.

Fig. 3(c) and Fig. 3(d) compare the total energy consumption and completion
time at different maximum CPU frequencies. Obviously, with the increase of
maximum CPU frequency, the energy consumption of the benchmark algorithm is
rising, which can be seen in Fig. 3(c). Intuitively, if the CPU frequency is
set at the maximum, the time consumption will be reduced. Therefore, in Fig.
3(d), the total time consumption is decreasing as CPU frequency increases.
However, the benchmark algorithm performs better than our algorithm only when
$w_{1}=0.9$ and $w_{2}=0.1$ after the maximum CPU frequency$=0.6$ GHz. The
reason is our algorithm mainly focuses on optimizing the energy when
$w_{1}=0.9$ and $w_{2}=0.1$, so our algorithm’s energy consumption is the
smallest in Fig. 3(c).

Besides, three lines related to our algorithm in Fig. 3(c) and Fig. 3(d) all
enter a stationary phase after a certain maximum CPU frequency, because the
optimal CPU frequency is already found for each device in the corresponding
range of CPU frequency. Thus, even if increasing the maximum CPU frequency,
the algorithm will find the previous optimal CPU frequency.

####  VII-B2 Different $\rho$ with fixed $(w_{1},w_{2})$

We fix the weight parameters $(w_{1},w_{2})=(0.5,0.5)$ of energy and time and
choose $\rho=1,~{}10,~{}20,~{}30,~{}40,~{}50,~{}60$, respectively, to compare
total energy and time consumption at different maximum computational
frequencies and transmission power limits. The results can be seen in Fig.
4(a), Fig. 4(b). MinPixel is the same as the benchmark algorithm in Fig. 3(a)
and Fig. 3(c). RandPixel randomly assigns the video frame resolution for each
user.

We plot Fig. 4(a) and Fig 4(b) by using logarithmic scale. It can be observed
that in Fig. 4(a) and Fig 4(b), both lines of energy consumption and time cost
of $\rho=1$ and $\rho=2$ are below the line of MinPixel. There is a noticeable
gap between our algorithm and MinPixel. In Fig. 4(a), energy consumption does
not change much versus the maximum transmission power. Since we optimize the
sum of energy and time, though energy consumption does not change much, the
total time decreases versus the maximum transmission power as shown in Fig.
4(b). Compared to MinPixel, our algorithm ($\rho=1$) reduces around 87% of the
total energy consumption and 25% of the total time. From $\rho=20$, the energy
and time consumption increase a lot. We analyze this is caused by the increase
of $\rho$. Larger $\rho$ means we emphasize the optimization of model
accuracy. Thus, the selection of video frame resolution will be $s_{\max}$ for
most devices, which results in a sharp increase in total energy and time
consumption after $\rho=20$. That is why the lines of
$\rho=20,~{}30,~{}40,~{}50$ are above MinPixel. Due to the same reason,
computational energy and time take a significant part of the total energy and
time consumption. Therefore, the lines of $\rho=20,~{}30,~{}40,~{}50$ appear
overlapping to each other. Nevertheless, our algorithm still improves about
71% of the total energy consumption and 43% of the total time.

Fig. 7 shows the relationship between $\rho$ and the average model accuracy.
We implement YOLOv5m [13], which is one of the object detection architectures
You Only Look Once (YOLO), on COCO dataset [12]. We implement the experiment
on a workstation with three NVIDIA RTX 2080 Ti GPUs for computation
acceleration. The YOLOv5m algorithm is modified to resize video frames
incoming to different resolutions, and object detection is then conducted
based on the resized frames. The purpose of this experiment is to evaluate the
performance of object detection in relation to video frame resolution.

(a) Total energy consumption with different $\rho$ under different maximum
transmit power limits. Here $(w_{1},w_{2})=(0.5,0.5)$.

(b) Total time with different $\rho$ under different maximum transmit power
limits. Here $(w_{1},w_{2})=(0.5,0.5)$.

Figure 4: Experiments with different $\rho$ under different maximum transmit
power limits.

Different from the above mentioned setting, we simulate 20 clients with one
aggregation server. The local epoch $R_{l}$ is 15 and the total aggregation
round $R_{g}$ is 50. Three lines in the figure correspond to
$s_{\min}=160,~{}256,~{}320$, respectively. The default $s_{\max}$ is still
640. As $s_{\min}$ increases, the starting model accuracy increases.
Generally, the metrics accuracy of analytics is defined as the ratio between
the number of objects that are correctly recognized and the total number of
objects in a video frame. Because the selection of $s_{\max}$ will result in
much more computational energy and time costs when $s_{\min}$ increases and
has already reached an acceptable model accuracy, the selection of $s_{\min}$
becomes the first choice. This is why when $\rho$ gets larger, the line of
$s_{\min}=160$ reaches the model accuracy of about $0.74$ first. At around
$\rho=15$, most devices will choose $s_{\max}=640$ as their video frame
resolution. While, for $s_{\min}=256,~{}320$, $\rho$ needs to reach $30,~{}40$
so that most devices select $s_{\max}=640$ as their video frame resolution.

Figure 5: Model accuracy under different $\rho$. The object detection is
implemented on the COCO dataset using modified YOLO algorithm. Here
$(w_{1},w_{2})=(0.5,0.5)$.

Figure 6: Total energy consumption versus maximum completion time. Here
$(w_{1},w_{2})=(1,0)$.

Figure 7: Total energy consumption versus maximum transmit power with fixed
maximum completion time $T_{1}=80$ s, $T_{2}=100$ s and $T_{3}=150$ s. Here
$(w_{1},w_{2})=(1,0)$. Total number of users is $N=50$.

###  VII-C Joint Communication & Computation Optimization vs.
Communication/Computation Optimization Only

To study the domination relationship between the transmission energy and
computation energy, we compare our algorithm with the following schemes at the
aspect of total energy consumption under different maximum completion times.
The maximum transmission power is $p^{\max}=10$ dBm. Because the maximum
completion time $\mathcal{T}$ is fixed to compare these three schemes, we set
$w_{1}=1$ and $w_{2}=0$ in our algorithm.

  * •

Communication optimization only: This scheme only optimizes communication
energy cost. Each device’s CPU frequency is set as a fixed value, and the
choice of video frame resolution is random. We only optimize the transmission
power and bandwidth allocated to each device. To guarantee there is a feasible
solution, the CPU frequency value is fixed for each device as
$\frac{R_{g}R_{l}\zeta sn^{2}c_{n}D_{n}}{T-R_{g}\max(d_{n}/r_{n})}$, which is
derived from constraint (14a), and $r_{n}$ is calculated from the initial
bandwidth and transmission power.

  * •

Computation optimization only: This scheme only optimizes computation energy
cost. Each device’s transmission power and bandwidth are fixed. The
transmission power and bandwidth of device $n$ are set as $p_{n}=p^{\max}$ and
$B_{n}=\frac{B}{2N}$.

Apparently, it can be observed from Fig. 7 that our proposed algorithm
performs better than the other two schemes. In addition, transmission energy
takes the lead in total energy consumption. Only optimizing the computation
energy cannot exceed the performance of only optimizing the transmission
energy. Furthermore, we can discover the conflicting relationship between
maximum completion time and total energy consumption. As maximum completion
time increases, there will be better solutions to reduce total energy
consumption. However, if we relax the constraint of the total delay
$\mathcal{T}$ too much, the optimal solution will reach the minimum limit of
transmission power and CPU frequency. That is why the gap between the lines of
Computation-only and our proposed algorithm becomes small when $\mathcal{T}$
is large.

###  VII-D Superiority of Our Optimization Algorithm over Existing Work

Yang _et al._ [7] has also proposed an optimization algorithm in the context
of federated learning by using FDMA. We name their method Scheme 1 in our
paper. Although Scheme 1 also has local accuracy as one of its optimization
variables, its local accuracy refers to the accuracy of stochastic gradient
descent algorithm and it is different from ours. Also, the optimization of the
local accuracy is a separate section in Scheme 1. Thus, it will not affect the
comparison between our method and Scheme 1. To be fair, we only use the method
(Algorithm 3 in [7]) they use to optimize the transmission power, the
bandwidth and the CPU frequency of each user.

The maximum completion time is not included in the objective function of the
optimization problem in Scheme 1. Instead, it appears as a constraint.
Therefore, to be fair, we set a fixed total completion time $\mathcal{T}$ to
compare our optimization algorithm with Scheme 1. With a fixed maximum
completion time $\mathcal{T}$, our minimization problem (14) is under the
condition that $w_{1}=1$ and $w_{2}=0$. Besides, the initial transmission
power and bandwidth of device $n$ are set as $p_{n}=p^{\max}$,
$B_{n}=\frac{B}{2N}$. The comparison result is shown in Fig. 7.

From Fig. 7, we can find that three green lines are below the corresponding
blue lines, which reveals that our algorithm’s performance is better.
Additionally, as the maximum completion time $\mathcal{T}$ decreases, the
total energy consumption gap between our algorithm and Scheme 1 gets larger,
thereby indicating that our proposed algorithm leads to a better solution when
the maximum completion time $\mathcal{T}$ is small. Therefore, our proposed
algorithm will be more appropriate for the training scenario with a restricted
training time.

At $T_{3}=150$ s, the solid line representing our algorithm first decreases
and then increases. This is because our initial transmission power is set as
$p_{n}=p^{\max}$. After finding the optimal solution at a specific $p^{\max}$,
the subsequent total energy consumption will slightly increase since the
different starting point from the previous step as $p^{\max}$ continues
increasing.

##  VIII Conclusion

Our paper considers an optimization problem for federated learning (FL) system
with mobile augmented reality (MAR) for the Metaverse. The optimization
variables include the bandwidth allocation, transmission power, CPU frequency
setting and MAR video frame resolution of each participating device. By
introducing three weight parameters, we optimize a weighted combination of the
total energy consumption, completion time and model accuracy. By adjusting the
weight parameters, our resource allocation scheme can be amenable to different
requirements of the FL-MAR system for the Metaverse. We present an algorithm
to solve the optimization problem via a rigorous analysis. The convergence and
computational complexity of the proposed algorithm are also provided. In the
experiments, we observe that our resource allocation algorithm outperforms the
existing schemes.

##
References

  * [1] X. Zhou, J. Zhao, H. Han, and C. Guet, “Joint Optimization of Energy Consumption and Completion Time in Federated Learning,” in _42nd International Conference on Distributed Computing Systems (ICDCS)_ , 2022.
  * [2] S.-M. Park and Y.-G. Kim, “A metaverse: Taxonomy, components, applications, and open challenges,” _IEEE_ , vol. 10, pp. 4209–4251, 2022.
  * [3] W. Zhang, B. Han, and P. Hui, “Jaguar: Low Latency Mobile Augmented Reality with Flexible Tracking,” in _Proceedings of the 26th ACM International Conference on Multimedia_ , ser. MM ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 355–363.
  * [4] K.-B. Park, S. H. Choi, M. Kim, and J. Y. Lee, “Deep Learning-Based Mobile Augmented Reality for Task Assistance Using 3D Spatial Mapping and Snapshot-Based RGB-D Data,” _Computers & Industrial Engineering_, vol. 146, p. 106585, 2020.
  * [5] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in _Artificial Intelligence and Statistics_. PMLR, 2017, pp. 1273–1282.
  * [6] J. D. N. Dionisio, W. G. B. III, and R. Gilbert, “3D virtual worlds and the metaverse: Current status and future possibilities,” _ACM Computing Surveys (CSUR)_ , vol. 45, no. 3, pp. 1–38, 2013.
  * [7] Z. Yang, M. Chen, W. Saad, C. S. Hong, and M. Shikh-Bahaei, “Energy Efficient Federated Learning over Wireless Communication Networks,” _IEEE Transactions on Wireless Communications_ , 2020.
  * [8] S. Luo, X. Chen, Q. Wu, Z. Zhou, and S. Yu, “Hfel: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learning,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 10, pp. 6535–6548, 2020.
  * [9] C. T. Dinh, N. H. Tran, M. N. Nguyen, C. S. Hong, W. Bao, A. Y. Zomaya, and V. Gramoli, “Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation,” _IEEE/ACM Transactions on Networking_ , 2020.
  * [10] P. S. Bouzinis, P. D. Diamantoulakis, and G. K. Karagiannidis, “Wireless Federated Learning (WFL) for 6G Networks–Part I: Research Challenges and Future Trends,” _arXiv preprint arXiv:2105.00842_ , 2021.
  * [11] Q. Liu, S. Huang, J. Opadere, and T. Han, “An Edge Network Orchestrator for Mobile Augmented Reality,” in _IEEE Conference on Computer Communications (INFOCOM)_. IEEE, 2018, pp. 756–764.
  * [12] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [13] G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, NanoCode012, Y. Kwon, TaoXie, K. Michael, J. Fang, imyhxy, Lorna, C. Wong, Z. Yifu, A. V, D. Montes, Z. Wang, C. Fati, J. Nadar, Laughing, UnglvKitDe, tkianai, yxNONG, P. Skalski, A. Hogan, M. Strobel, M. Jain, L. Mammana, and xylieong, “ultralytics/yolov5: v6.2 - YOLOv5 Classification Models, Apple M1, Reproducibility, ClearML and Deci.ai integrations,” Aug. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.7002879
  * [14] Y. Jong, “An Efficient Global Optimization Algorithm for Nonlinear Sum-of-Ratios Problem,” _Optimization Online_ , 2012.
  * [15] D. Chatzopoulos, C. Bermejo, Z. Huang, and P. Hui, “Mobile Augmented Reality Survey: From Where We are to Where We Go,” _IEEE Access_ , vol. 5, pp. 6917–6950, 2017.
  * [16] E. Bottani and G. Vignali, “Augmented Reality Technology in the Manufacturing Industry: A Review of the Last Decade,” _IISE Transactions_ , vol. 51, no. 3, pp. 284–310, 2019.
  * [17] H. Wang, B. Kim, J. Xie, and Z. Han, “E-Auto: A Communication Scheme for Connected Vehicles with Edge-Assisted Autonomous Driving,” in _IEEE International Conference on Communications (ICC)_. IEEE, 2019, pp. 1–6.
  * [18] H. Wang and J. Xie, “User preference based energy-aware mobile AR system with edge computing,” in _IEEE INFOCOM 2020 - IEEE Conference on Computer Communications_ , 2020, pp. 1379–1388.
  * [19] Q. Liu and T. Han, “Dare: Dynamic adaptive mobile augmented reality with edge computing,” in _2018 IEEE 26th International Conference on Network Protocols (ICNP)_. IEEE, 2018, pp. 1–11.
  * [20] P. Lin, Q. Song, D. Wang, F. R. Yu, L. Guo, and V. C. Leung, “Resource Management for Pervasive-Edge-Computing-Assisted Wireless VR Streaming in Industrial Internet of Things,” _IEEE Transactions on Industrial Informatics_ , vol. 17, no. 11, pp. 7607–7617, 2021.
  * [21] H. Zhang, G. Ananthanarayanan, P. Bodik, M. Philipose, P. Bahl, and M. J. Freedman, “Live Video Analytics at Scale with Approximation and Delay-Tolerance,” in _14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17)_ , 2017, pp. 377–392.
  * [22] Y. Zhou, C. Pan, P. L. Yeoh, K. Wang, M. Elkashlan, B. Vucetic, and Y. Li, “Communication-and-Computing Latency Minimization for UAV-Enabled Virtual Reality Delivery Systems,” _IEEE Transactions on Communications_ , vol. 69, no. 3, pp. 1723–1735, 2020.
  * [23] J. Ahn, J. Lee, S. Yoon, and J. K. Choi, “A Novel Resolution and Power Control Scheme for Energy-Efficient Mobile Augmented Reality Applications in Mobile Edge Computing,” _IEEE Wireless Communications Letters_ , vol. 9, no. 6, pp. 750–754, 2019.
  * [24] L. N. Huynh, Y. Lee, and R. K. Balan, “Deepmon: Mobile Gpu-Based Deep Learning Framework for Continuous Vision Applications,” in _International Conference on Mobile Systems, Applications, and Services_ , 2017, pp. 82–95.
  * [25] S. Han, H. Shen, M. Philipose, S. Agarwal, A. Wolman, and A. Krishnamurthy, “Mcdnn: An Approximation-Based Execution Framework for Deep Stream Processing under Resource Constraints,” in _Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services_ , 2016, pp. 123–136.
  * [26] S. Jiang, Z. Lin, Y. Li, Y. Shu, and Y. Liu, “Flexible High-Resolution Object Detection on Edge Devices with Tunable Latency,” in _International Conference on Mobile Computing and Networking_ , 2021, pp. 559–572.
  * [27] H. Park, Y. Lee, and J. Ko, “Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras,” _ACM Conf on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 5, no. 2, pp. 1–30, 2021.
  * [28] J. Kang, D. Ye, J. Nie, J. Xiao, X. Deng, S. Wang, Z. Xiong, R. Yu, and D. Niyato, “Blockchain-based federated learning for industrial metaverses: Incentive scheme with optimal aoi,” in _IEEE International Conference on Blockchain_ , 2022.
  * [29] S. Zeng, Z. Li, H. Yu, Z. Zhang, L. Luo, B. Li, and D. Niyato, “Hfedms: Heterogeneous federated learning with memorable data semantics in industrial metaverse,” 2022. [Online]. Available: https://arxiv.org/abs/2211.03300
  * [30] M. Xu, D. Niyato, J. Kang, Z. Xiong, C. Miao, and D. I. Kim, “Wireless edge-empowered metaverse: A learning-based incentive mechanism for virtual reality,” in _IEEE International Conference on Communications_ , 2022, pp. 5220–5225.
  * [31] P. Si, J. Zhao, H. Han, K.-Y. Lam, and Y. Liu, “Resource allocation and resolution control in the metaverse with mobile augmented reality,” _arXiv preprint arXiv:2209.13871_ , 2022.
  * [32] Q. Zeng, Y. Du, K. Huang, and K. K. Leung, “Energy-Efficient Radio Resource Allocation for Federated Edge Learning,” in _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_. IEEE, 2020, pp. 1–6.
  * [33] D. Chen, L. J. Xie, B. Kim, L. Wang, C. S. Hong, L.-C. Wang, and Z. Han, “Federated Learning Based Mobile Edge Computing for Augmented Reality Applications,” in _2020 International Conference on Computing, Networking and Communications (ICNC)_ , 2020, pp. 767–773.
  * [34] C. E. Shannon, “Communication in the presence of noise,” _Proceedings of the IRE_ , vol. 37, no. 1, pp. 10–21, 1949.
  * [35] Y. Mao, J. Zhang, and K. B. Letaief, “Dynamic Computation Offloading for Mobile-Edge Computing with Energy Harvesting Devices,” _IEEE Journal on Selected Areas in Communications_ , vol. 34, no. 12, pp. 3590–3605, 2016.
  * [36] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 779–788.
  * [37] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet Classification with Deep Convolutional Neural Networks,” _Advances in Neural Information Processing Systems_ , vol. 25, 2012.
  * [38] K. He and J. Sun, “Convolutional Neural Networks at Constrained Time Cost,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 5353–5360.
  * [39] H. Wang and J. Xie, “User Preference Based Energy-Aware Mobile AR System with Edge Computing,” in _IEEE Conference on Computer Communications (INFOCOM)_ , 2020, pp. 1379–1388.
  * [40] X. Yu, D. Xu, and R. Schober, “Enabling Secure Wireless Communications via Intelligent Reflecting Surfaces,” in _2019 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2019, pp. 1–6.
  * [41] M. Grant and S. Boyd, “CVX: Matlab Software for Disciplined Convex Programming, version 2.1,” 2014.
  * [42]
  * [43] S. He, Y. Huang, L. Yang, and B. Ottersten, “Coordinated Multicell Multiuser Precoding for Maximizing Weighted Sum Energy Efficiency,” _IEEE Transactions on Signal Processing_ , vol. 62, no. 3, pp. 741–751, 2013.
  * [44] Z.-Q. Luo, W.-K. Ma, A. M.-C. So, Y. Ye, and S. Zhang, “Semidefinite Relaxation of Quadratic Optimization Problems,” _IEEE Signal Processing Magazine_ , vol. 27, no. 3, pp. 20–34, 2010.
  * [45] J. Zheng, W. Ni, and H. Tian, “QoS-Constrained Federated Learning Empowered by Intelligent Reflecting Surface,” _arXiv preprint arXiv:2103.11551_ , 2021\.

##  Appendix A Notations

TABLE II: Notations Description | Notation
---|---
The number of total devices | $N$
Total energy consumption | $\mathcal{E}$
The completion time of the whole federated learning process | $\mathcal{T}$
The accuracy metric | $\mathcal{A}$
Maximum uplink bandwidth | $B$
Video frame resolution of device $n$ | $s_{n}$
Allocated bandwidth of device $n$ | $B_{n}$
The number of samples on device $n$ | $D_{n}$
Total number of samples on all devices | $D$
Transmission data size of user device $n$ | $d_{n}$
Maximum completion time between two global aggregations | $T$
Data transmission time of device $n$ | $T^{up}_{n}$
Local computation time of user device $n$ | $T^{cmp}_{n}$
Data transmission rate of device $n$ | $r_{n}$
Minimum transmission rate of device $n$ | $r_{n}^{\min}$
CPU frequency of device $n$ | $f_{n}$
Minimum and maximum CPU frequency of device $n$ | $f_{n}^{\min},f_{n}^{\max}$
The number of local iterations in one global round | $R_{l}$
The number of global rounds | $R_{g}$
The effective switched capacitance | $\kappa$
CPU cycles per standard sample of device $n$ | $c_{n}$
Local computation energy consumption of user device $n$ | $E_{n}^{cmp}$
Transmission energy consumption of device $n$ | $E_{n}^{trans}$
Weight parameters |  $w_{1}$, $w_{2}$, $\rho$
Noise power spectral density | $N_{0}$
Transmission power of device $n$ | $p_{n}$
Minimum and maximum transmission power of device $n$ | $p_{n}^{\min},p_{n}^{\max}$
Channel gain from device $n$ to the base station | $g_{n}$
Lagrange multipliers | $\bm{\lambda},\bm{\tau},\mu$
Auxiliary variables | $\bm{\nu},\bm{\beta}$

##  Appendix B Solution to Subproblem 1

We treat the optimization problem by the Karush–Kuhn–Tucker (KKT) approach and
introduce the Lagrange function

$\displaystyle L_{1}(f_{n},\hat{s_{n}},T,\bm{\lambda})$
$\displaystyle\\!=\\!w_{1}R_{g}\sum_{n=1}^{N}\kappa
R_{l}\zeta\hat{s_{n}}^{2}c_{n}D_{n}f_{n}^{2}\\!+\\!w_{2}R_{g}T\\!-\\!\rho\sum_{n=1}^{N}\hat{A}_{n}(\hat{s_{n}})$
$\displaystyle\\!+\\!\sum_{n=1}^{N}\lambda_{n}[(\frac{R_{l}\zeta\hat{s_{n}}^{2}c_{n}D_{n}}{f_{n}}\\!+\\!T_{n}^{up})\\!-\\!T],$
(A.1)

where $\bm{\lambda}:=[\lambda_{1},\lambda_{2},\ldots,\lambda_{N}]$ is the
Lagrange multiplier associated with the inequality constraint (14a).

After applying KKT conditions to (B), we get

$\displaystyle\frac{\partial L_{1}}{\partial
f_{n}}=2w_{1}R_{g}R_{l}\kappa\zeta\hat{s_{n}}^{2}c_{n}D_{n}f_{n}-\lambda_{n}\frac{R_{l}\zeta\hat{s_{n}}^{2}c_{n}D_{n}}{f_{n}^{2}}=0,~{}\forall
n\in\mathcal{N},$ (A.2) $\displaystyle{\frac{\partial
L_{1}}{\partial\hat{s_{n}}}=2w_{1}R_{g}R_{l}\kappa\zeta\hat{s_{n}}c_{n}D_{n}f_{n}^{2}+\lambda_{n}\frac{2R_{l}\zeta\hat{s_{n}}c_{n}D_{n}}{f_{n}}}-\rho\hat{k}_{n}=0,~{}\forall
n\in\mathcal{N},$ (A.3) $\displaystyle\frac{\partial L_{1}}{\partial
T}=w_{2}R_{g}-\sum_{n}^{N}\lambda_{n}=0,~{}\lambda_{n}[(\frac{R_{l}\zeta\hat{s_{n}}^{2}c_{n}D_{n}}{f_{n}}+T_{n}^{up})-T]=0,$
(A.4)

from which we derive

$\displaystyle f_{n}^{*}$
$\displaystyle=\sqrt[3]{\frac{\lambda_{n}}{2w_{1}R_{g}\kappa}},~{}\hat{s_{n}}^{*}=\frac{\rho\hat{k}_{n}}{2R_{l}\zeta
c_{n}D_{n}\cdot[w_{1}R_{g}\kappa(f_{n}^{*})^{2}+\frac{\lambda_{n}}{f_{n}^{*}}]},$
(A.5) $\displaystyle w_{2}$
$\displaystyle=\frac{\sum_{n}^{N}\lambda_{n}}{R_{g}}.$ (A.6)

Through using Eq. (A.5), we can use $\lambda_{n}$ to represent $f_{n}$ and
$\hat{s_{n}}$. Then, the dual problem is

$\displaystyle\max_{\lambda_{n}}~{}$
$\displaystyle\sum_{n=1}^{N}-\frac{\rho^{2}\hat{k}_{n}^{2}}{4h(2^{-\frac{2}{3}}+2^{\frac{1}{3}})}\lambda_{n}^{-\frac{2}{3}}+T_{n}^{up}\lambda_{n}+\rho\hat{k}_{n}s_{\min}-\rho
A_{n}(s_{\min})$ (A.7) $\displaystyle\text{subject to},$
$\displaystyle\sum_{n=1}^{N}\lambda_{n}=w_{2}R_{g},~{}\lambda_{n}\geq 0,$
(A.7a)

where $h=R_{l}\zeta c_{n}D_{n}(w_{1}\kappa R_{g})^{\frac{1}{3}}$. Obviously,
this dual problem is a simple convex optimization problem. In this paper, we
use CVX [41] to solve it and get the optimal
$\bm{\lambda^{*}}=[\lambda_{1}^{*},...,\lambda_{n}^{*}]$. Then, we are able to
calculate $\bm{f}$ and $\bm{\hat{s}}$ through Eq. (A.5).

##  Appendix C Proof of Lemma 1

It is obvious that $K(p_{n},B_{n})=p_{n}d_{n}$ is an affine function with
$B_{n}=0$. Thus, $K(p_{n},B_{n})$ is convex.

To prove $G(p_{n},B_{n})$ is concave, we first calculate its Hessian.

$\displaystyle
Hessian(G)=\begin{bmatrix}-\frac{g_{n}^{2}}{B_{n}N_{0}^{2}(\frac{g_{n}p_{n}}{B_{n}N_{0}}+1)^{2}}&\frac{g_{n}^{2}p_{n}}{B_{n}^{2}N_{0}^{2}(\frac{g_{n}p_{n}}{B_{n}N_{0}}+1)^{2}}\\\\[-5.0pt]
\frac{g_{n}^{2}p_{n}}{B_{n}^{2}N_{0}^{2}(\frac{g_{n}p_{n}}{B_{n}N_{0}}+1)^{2}}&-\frac{g_{n}^{2}p_{n}^{2}}{B_{n}^{3}N_{0}^{2}(\frac{g_{n}p_{n}}{B_{n}N_{0}}+1)^{2}}\end{bmatrix}.$
(A.8)

Given a vector $\bm{x}=[x_{1},x_{2}]^{T}$ in $\mathbb{R}^{2}$, we have

$\displaystyle\bm{x}^{T}Hessian(G)\bm{x}=-\frac{(x_{1}g_{n}B_{n}-x_{2}g_{n}p_{n})^{2}}{B_{n}^{3}N_{0}^{2}(\frac{g_{n}p_{n}}{B_{n}N_{0}}+1)^{2}}\leq
0.$ (A.9)

Because $\bm{x}^{T}*Hessian(G)*\bm{x}\leq 0$ for all $\bm{x}$ in
$\mathbb{R}^{2}$, $Hessian(G)$ is a negative semidefinite matrix. Therefore,
$G(p_{n},B_{n})$ is a concave function. Lemma 1 is proved.

##  Appendix D Proof of Theorem 2

Before applying KKT conditions, we write down the partial Lagrangian function
of problem SP2_v2:

$\displaystyle L_{2}(p_{n},B_{n},\tau_{n},\mu)=$
$\displaystyle\sum_{n=1}^{N}\nu_{n}(p_{n}d_{n}-\beta_{n}B_{n}\log_{2}(1+\frac{p_{n}g_{n}}{N_{0}B_{n}}))-\sum_{n=1}^{N}\tau_{n}(B_{n}\log_{2}(1+\frac{p_{n}g_{n}}{N_{0}B_{n}})-r_{n}^{\min})+\mu(\sum_{n=1}^{N}B_{n}-B),$
(A.10)

where $\tau_{n}|_{n=1,\ldots,N}$ and $\mu$ are non-negative Lagrange
multipliers.

After applying KKT conditions to problem SP2_v2, we get

$\displaystyle\frac{\partial L_{2}}{\partial p_{n}}$
$\displaystyle=\nu_{n}(d_{n}-\frac{\beta_{n}g_{n}}{N_{0}(1+\vartheta_{n})\ln
2})-\frac{\tau_{n}g_{n}}{N_{0}(1+\vartheta_{n})\ln 2}=0,$ (A.11)
$\displaystyle\frac{\partial L_{2}}{\partial B_{n}}$
$\displaystyle=-(\nu_{n}\beta_{n}+\tau_{n})\log_{2}(1+\vartheta_{n})+\frac{(\nu_{n}\beta_{n}+\tau_{n})p_{n}g_{n}}{(1+\vartheta_{n})\ln
2N_{0}B_{n}}+\mu=0,$ (A.12) $\displaystyle-\tau_{n}$
$\displaystyle(B_{n}\log_{2}(1+\vartheta_{n})-r_{n}^{\min})=0,$ (A.13)
$\displaystyle\mu(\sum_{n=1}^{N}$ $\displaystyle B_{n}-B)=0,$ (A.14)

where $\vartheta_{n}=\frac{p_{n}g_{n}}{N_{0}B_{n}}$ for $n=1,\cdots,N$.

We derive the relationship between $\bm{p}$ and $\bm{B}$ from (A.11), so

$\displaystyle
p_{n}=(\frac{(\nu_{n}\beta_{n}+\tau_{n})g_{n}}{N_{0}d_{n}\nu_{n}\ln
2}-1)\frac{N_{0}B_{n}}{g_{n}}$ (A.15)

From (A.13) and (A.15), we could analyze

$\displaystyle
B_{n}=\frac{r_{n}^{\min}}{\log_{2}(\frac{(\nu_{n}\beta_{n}+\tau_{n})g_{n}}{N_{0}d_{n}\nu_{n}\ln
2})},\text{~{}if~{}}\tau_{n}\neq 0.$ (A.16)

Substituting (A.15) in (A.10), we replace $p_{n}$ by $B_{n}$ in the Lagrangian
function:

$\displaystyle L_{3}(B_{n},$
$\displaystyle\tau_{n},\mu)=B_{n}(\frac{\nu_{n}\beta_{n}+\tau_{n}}{\ln
2}-\frac{\nu_{n}d_{n}N_{0}}{g_{n}}-(\nu_{n}\beta_{n}+\tau_{n})\times\log_{2}(\frac{(\nu_{n}\beta_{n}+\tau_{n})g_{n}}{N_{0}d_{n}\nu_{n}\ln
2})+\mu)+\sum_{n=1}^{N}r_{n}^{\min}*\tau_{n}-\mu B.$ (A.17)

The corresponding dual problem writes as

$\displaystyle\max_{\tau_{n},\mu}~{}$ $\displaystyle
g(\tau_{n},\mu)=\sum_{n=1}^{N}r_{n}^{\min}*\tau_{n}-\mu B$ (A.18) subject to
$\displaystyle\frac{a_{n}}{\ln 2}-j_{n}-a_{n}\log_{2}(\frac{a_{n}}{j_{n}\ln
2})+\mu=0,$ (A.19) $\displaystyle\mu\geq 0,~{}\tau_{n}\geq 0,$ (A.20)

where $a_{n}=\nu_{n}\beta_{n}+\tau_{n}$ and
$j_{n}=\frac{\nu_{n}d_{n}N_{0}}{g_{n}}$. Besides, constraint (A.19) derives
the relationship between Lagrange multipliers $\tau_{n}$ and $\mu$:

$\displaystyle\tau_{n}=\frac{(\mu-j_{n})\ln 2}{W(\frac{\mu-
j_{n}}{e*j_{n}})}-\nu_{n}\beta_{n},$ (A.21)

where $W(\cdot)$ is Lambert $W$ function and $e$ is the base of the natural
logarithms.

Given (A.21), the dual function is simplified as

$\displaystyle g(\mu)=\sum_{n=1}^{N}r_{n}^{\min}*(\frac{(\mu-j_{n})\ln
2}{W(\frac{\mu-j_{n}}{e*j_{n}})}-\nu_{n}\beta_{n})-\mu B.$ (A.22)

Take the first derivative and we get
$g^{\prime}(\mu)=\sum_{n=1}^{N}\frac{r_{n}^{\min}\ln 2}{W(\frac{\mu-
j_{n}}{e*j_{n}})}(1-\frac{1}{W(\frac{\mu-j_{n}}{e*j_{n}})+1})-B$.
Additionally, $g^{\prime\prime}(\mu)=-\sum_{n=1}^{N}\frac{r_{n}^{\min}\ln
2W(\frac{\mu-c_{n}}{e*c_{n}})}{(\mu-j_{n})(1+W(\frac{\mu-
j_{n}}{e*j_{n}}))^{3}}\leq 0$ because $\frac{\mu-
j_{n}}{e*j_{n}}\geq-\frac{1}{e}$.

Therefore, $g^{\prime}(\mu)$ is a monotone decreasing function, and $g(\mu)$
is concave, so $g(\mu)$ reaches maximum when $g^{\prime}(\mu)=0$. The
bisection method can be used to find $\mu$ satisfying $g^{\prime}(\mu)=0$.
Naturally, $\tau_{n}=\max((\ref{tau_mu_relation}),0)$. Note $\tau_{n}\neq 0$
implies
$B_{n}=\frac{r_{n}^{\min}}{\log_{2}(\frac{(\nu_{n}\beta_{n}+\tau_{n})g_{n}}{N_{0}d_{n}\nu_{n}\ln
2})}$. We denote the sum of bandwidth of these devices by $B_{\tau_{n}\neq 0}$
and the number of these devices by $N_{\tau_{n}\neq 0}$

Then, replacing $p_{n}$ by $B_{n}$ according to (A.15) in problem SP2_v2,
removing those devices whose bandwidth is calculated in the previous step and
considering the remaining devices, the new problem becomes

$\displaystyle\min_{B_{n}}~{}$ $\displaystyle\sum_{n=1}^{N-N_{\tau_{n}\neq
0}}(\frac{\nu_{n}\beta_{n}}{\ln
2}-\frac{N_{0}d_{n}\nu_{n}}{g_{n}}-\nu_{n}\beta_{n}\times\log_{2}(\frac{\beta_{n}g_{n}}{N_{0}d_{n}\ln
2}))B_{n}$ (A.23) $\displaystyle\text{subject to},$ $\displaystyle
p_{n}^{\min}\leq(\frac{\beta_{n}g_{n}}{N_{0}d_{n}\ln
2}-1)\frac{N_{0}B_{n}}{g_{n}}\leq p_{n}^{\max},~{}\forall
n\in\mathcal{N}\backslash\mathcal{N}_{\tau_{n}\neq 0},$ (A.24)
$\displaystyle\sum_{n=1}^{N-N_{\tau_{n}\neq 0}}B_{n}\leq B-B_{\tau_{n}\neq
0},$ (A.25)

where $\mu$ and $\tau_{n}$ have already been solve in previous steps.

Until now, the problem SP2_v2 has become a more simple convex optimization
problem with just one variable. Thus, we can use the convex problem solver CVX
[41] to solve problem (A.23).